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ABSTRACT 


The  U.S.  Congress  passed  Public  Law  1 11-23  (Weapon  Systems  Acquisition  Reform  Act 
(WSARA)  of  2009),  which  significantly  changed  the  way  the  U.S.  Department  of  Defense 
(DoD)  procures  weapon  systems.  The  WSARA  of  2009  increases  the  role  of  the  Office  of 
Secretary  of  Defense  (OSD)  in  the  Services’  Analyses  of  Alternatives  (AoAs)  through  the 
creation  of  the  Director,  Cost  Assessment  and  Program  Evaluation  (CAPE).  Before  Major 
Defense  Acquisition  Programs  (MDAPs)  Milestone  (MS)  Decisions,  AoAs  are  performed  to 
detennine  if  an  acquisition  program  is  warranted  to  start  or  to  continue  to  the  next  acquisition 
phase.  The  U.S.  Anny  has  performed  several  AoAs  for  Acquisition  Category  (ACAT)  I 
programs  since  the  passing  of  the  WSARA  of  2009.  These  AoAs  have  had  varying  degrees  of 
success  in  meeting  the  expectations  of  OSD  CAPE. 

This  report  outlines  the  research  to  identify  potential  ways  to  improve  U.S.  Anny  AoAs 
to  better  meet  the  requirements  of  the  WSARA  of  2009  and  the  expectations  of  OSD  CAPE.  Key 
senior-level  Department  of  the  Anny  and  OSD  CAPE  personnel  associated  with  recently 
conducted  ACAT  I  AoAs  were  interviewed,  and  their  responses  are  presented  in  this  report. 
Recommendations  are  provided  on  key  aspects  of  U.S.  Army  AoAs  to  include  the  following:  the 
major  issues  associated  with  AoAs  under  the  WSARA  of  2009,  the  relationship  between  OSD 
CAPE  and  the  Anny,  the  timing  of  conducting  AoAs  and  other  analyses,  and  the  role  of 
alternatives  in  AoAs.  Special  consideration  was  given  to  addressing  the  role  of  foreign 
nondevelopmental  items  in  AoAs  and  the  tradeoffs  among  cost,  schedule,  and  performance. 
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CHAPTER  1 
INTRODUCTION 


This  report  outlines  the  research  to  determine  how  to  improve  U.S.  Anny  Analyses  of 
Alternatives  (AoAs)  to  better  meet  the  requirements  of  the  Weapon  Systems  Acquisition  Refonn 
Act  (WSARA)  of  2009  and  the  expectations  of  the  Office  of  Secretary  of  Defense  (OSD) 
Director,  Cost  Assessment  and  Program  Evaluation  (CAPE).  Key  senior-level  Department  of  the 
Anny  and  OSD  CAPE  personnel  associated  with  recently  conducted  Acquisition  Category 
(ACAT)  I  AoAs  were  interviewed,  and  their  responses  are  presented  in  this  report.  The  report 
also  provides  recommendations  to  be  made  to  the  process. 

Background 

Before  Major  Defense  Acquisition  Programs  (MDAPs)  Milestone  (MS)  Decisions,  AoAs 
are  perfonned  to  detennine  if  an  acquisition  program  is  warranted  to  start  or  to  continue  to  the 
next  acquisition  phase.  AoAs  are  statutory  requirements  for  MS  decisions  based  on  Department 
of  Defense  Instruction  (DoDI)  5000.02  (2008),  United  States  Code  (U.S.C.),  Title  10  (2011)  and 
U.S.C.  Title  40  (2011).  In  2009,  the  U.S.  Congress  passed  Public  Law  111-23  (WSARA  of 
2009),  which  significantly  changed  the  way  DoD  procures  weapon  systems.  The  WSARA  of 
2009  increases  the  role  of  the  OSD  for  AoAs  by  the  creation  of  OSD  CAPE.  The  organization  of 
OSD  CAPE  must  develop  AoA  guidance,  and  the  Materiel  Decision  Authority  (MDA)  must 
certify  that  an  AoA  has  been  completed  based  on  this  guidance  (WSARA  of  2009).  This 
increased  scrutiny  on  the  Services’  AoAs  has  made  it  critical  that  the  U.S.  Anny  perform  the 
AoAs  consistent  with  OSD  CAPE  guidance.  The  WSARA  of  2009  also  has  placed  additional 
emphasis  on  tradeoffs  between  cost,  schedule,  and  perfonnance  in  AoAs. 

Problem  Statement 

The  U.S.  Anny  has  perfonned  several  AoAs  for  ACAT  I  programs  since  the  passing  of 
the  WSARA  of  2009.  These  AoAs  had  varying  degrees  of  success  in  meeting  the  expectations  of 
OSD  CAPE. 

Purpose  of  this  Study 

This  study  will  identify  potential  improvements  for  U.S.  Anny  AoAs  to  better  meet  the 
requirements  of  the  WSARA  of  2009  and  the  expectations  of  OSD  CAPE.  Recommendations  are 
provided  on  key  aspects  of  U.S.  Anny  AoAs  to  include  the  following:  the  major  issues 
associated  with  AoAs  under  the  WSARA  of  2009,  the  relationship  between  OSD  CAPE  and  the 
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Army,  the  timing  of  conducting  AoAs  and  other  analyses,  and  the  role  of  alternatives  in  AoAs. 
Special  consideration  will  be  given  to  addressing  the  role  of  foreign  nondevelopmental  items 
(NDIs)  in  AoAs  and  the  tradeoffs  among  cost,  schedule,  and  performance. 

Significance  of  this  Research 

This  research  will  help  identify  ways  to  improve  U.S.  Army  AoAs  to  better  meet 
expectations  of  OSD  CAPE  in  order  for  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  (USD(AT&L))  to  certify  that  an  AoA  has  been  completed.  Further, 
efficiency  and  effectiveness  are  improved  with  better  AoAs.  In  the  expected  post-Operation 
Enduring  Freedom  Defense  budget  restrictions,  the  importance  of  quality  AoAs  is  even  more 
paramount.  AoAs  are  needed  for  an  MS  A  decision  and  potentially  for  MS  B  and  MS  C 
decisions.  Without  a  certified  AoA,  the  program  cannot  go  to  a  MS  decision.  The  AoA  must  be 
performed  consistent  with  the  guidance  developed  by  OSD  CAPE  (WSARA  of  2009)  for  MS  A 
or  any  other  MS. 

Overview  of  the  Research  Methodology 

This  research  utilized  the  interview  method  to  collect  data  from  senior-level  Army 
leadership  involved  with  AoAs  and  senior-level  OSD  CAPE  personnel.  The  Anny  organizations 
were  divided  into  two  groups;  including  Army  AoA  team  members  and  Anny  AoA  oversight. 
Senior-level  personnel  are  defined  as  Senior  Executive  Service  (SES)  personnel,  colonels 
(COLs)  (or  0-6  equivalents),  and  General  Schedule  (GS)-15  (or  equivalents).  These  senior-level 
personnel  have  been  involved  with  U.S.  Army  AoAs  since  the  passing  of  WSARA  of  2009. 
Their  involvement  may  have  been  developing  guidance,  reviewing,  approving,  or  certifying  U.S. 
Anny  AoAs. 

Research  Question 

How  can  Army  AoAs  be  improved  to  better  meet  the  expectations  of  OSD  CAPE  under 
WSARA  of  2009? 

Research  Hypotheses 

There  are  two  hypotheses  for  this  research,  which  include  the  following: 

1.  U.S.  Anny  AoAs  can  be  improved  to  better  meet  the  expectations  of  the  OSD  CAPE 
under  the  WSARA  of  2009. 

2.  Senior-level  Army  and  OSD  CAPE  personnel  can  identify  potential  improvements  to 
Anny  AoAs  to  better  meet  WSARA  of  2009  requirements  and  OSD  CAPE  expectations. 
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Limitations  of  the  Study 

This  study  was  limited  to  U.S.  Army  AoAs  for  ACAT  I  programs.  Other  Service  AoAs, 
or  AoAs  associated  with  ACAT  II,  III,  or  IV  programs,  are  important  but  out  of  the  scope  of  this 
research.  This  research  was  also  limited  by  the  number  of  people  interviewed.  Due  to  the 
constraints  of  completing  the  research  by  April  12,  2012,  a  selected  group  of  senior-level  U.S. 
Anny  and  OSD  CAPE  individuals  were  interviewed. 

Timeline  for  Research 

The  following  are  the  key  milestones  for  the  completion  of  this  research  by  the  due  date 
of  April  12,2012. 

October  14,  2011 — Sent  draft  research  proposal  to  the  project  advisers  and  Mr.  James 
Oman,  Director,  Aberdeen  Proving  Ground  Senior  Service  College  Fellowship 
Program. 

November  30,  2011 — Met  with  Mr.  Robert  Chandler,  Chief,  Weapons  Systems  Analysis 
Division,  Army  Materiel  Systems  Analysis  Activity  (AMSAA),  to  discuss  research 
proposal. 

November  30,  2011 — Met  with  Mr.  Patrick  O’Neill,  Technical  Director,  AMSAA,  to 
discuss  research  proposal. 

December  15,  2011 — Met  with  Dr.  William  Forrest  Crain,  Director,  AMSAA,  to  discuss 
research  proposal. 

December  31,  2011 — Performed  mock  interviews  with  AMSAA  personnel  to  refine 
questions. 

January  9,  2012 — Sent  final  research  proposal  to  project  advisers. 

February  26,  2012 — Sent  first  draft  Strategic  Research  Project  to  project  advisers. 

March  16,  2012 — Completed  interviews. 

March  26,  2012 — Sent  second  draft  Strategic  Research  Project  to  project  advisers. 

April  12,  2012 — Final  Strategic  Research  Project  sent  to  project  advisers. 

Because  of  scheduling  issues  associated  with  the  interviews,  some  interviews  were 
conducted  after  the  March  16,  2012  deadline.  The  last  interview  was  conducted  on  April  2,  2012. 
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CHAPTER  2 
LITERATURE  REVIEW 


This  chapter  provides  the  following:  an  overview  of  AoAs;  an  explanation  of  how  the 
AoA  process  has  changed  under  the  WSARA  of  2009;  an  overview  how  DoD  is  implementing 
the  WSARA  of  2009;  U.S.  Army  initiatives  to  improve  AoAs;  and  OSD  CAPE  suggestions  for 
AoAs. 

AoA  Overview 

AoAs  are  a  required  part  in  the  DoD  acquisition  process  as  prescribed  in  Department  of 
Defense  Instruction  (DoDI)  5000.02  (2008),  U.S.C.  Title  10,  and  U.S.C.  Title  40.  DoDI  5000.02 
(2008)  requires  that  AoA  guidance  be  developed  at  the  Materiel  Development  Decision  (MDD) 
review  by  the  Director,  Program  Analysis  &  Evaluation  (DPA&E).  The  WSARA  of  2009 
created  a  new  OSD  office  called  OSD  CAPE  with  a  political  appointee.  OSD  CAPE  was 
designated  to  be  the  lead  for  the  development  of  AoA  guidance.  The  lead  DoD  component 
prepares  an  AoA  study  plan  to  assess  materiel  solutions,  identify  key  technologies,  and  estimate 
life  cycle  costs  (DoDI  5000.02,  2008).  “The  purpose  of  the  AoA  is  to  assess  the  potential 
materiel  solutions  to  satisfy  the  capability  need  documented  in  the  approved  ICD  [Initial 
Capabilities  Document]”  (DoDI  5000.02,  2008). 

DoDI  5000.02  (2008)  provides  additional  guidance  on  AoAs  for  the  Materiel  Solutions 
Analysis  (MSA)  Phase: 

The  AoA  shall  focus  on  identification  and  analysis  of  alternatives, 
measures  of  effectiveness,  cost,  schedule,  concepts  of  operations,  and  overall 
risk.  The  AoA  shall  assess  the  critical  technology  elements  (CTEs)  associated 
with  each  proposed  materiel  solution,  including  technology  maturity,  integration 
risk,  manufacturing  feasibility,  and,  where  necessary,  technology  maturation 
and  demonstration  needs.  To  achieve  the  best  possible  system  solution, 
emphasis  shall  be  placed  on  innovation  and  competition.  Existing  commercial- 
off-the-shelf  (COTS)  functionality  and  solutions  drawn  from  a  diversified  range 
of  large  and  small  businesses  shall  be  considered  (DoDI  5000.02,  2008,  p.  15). 

The  MSA  Phase  ends  when  the  AoA  has  been  completed,  and  entrance  into  the 
Technology  Development  (TD)  phase  is  dependent  on  the  completion  of  the  AoA.  DoDI  5000.02 
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(2008)  states  that  AoAs  are  a  statutory  requirements  for  MS  A,  MS  B  (updated  as  necessary), 
and  MS  C  (updated  as  necessary)  based  on  U.S.C.  Title  10  (2011)  and  U.S.C.  Title  40  (2011).  In 
addition  to  fulfilling  a  statutory  requirement  for  MS  decisions,  AoAs  must  also  meet  other 
requirements  in  U.S.C.  Title  40  (2011)  including  process  improvement  to  reduce  costs,  improve 
effectiveness,  maximize  the  use  of  COTS  equipment,  and  detennine  that  no  private  sector  or 
government  organization  can  better  support  the  function  (DoDI  5000.02,  2008).  OSD  will  ensure 
that  AoAs  have: 

a.  illuminated  capability  advantages  and  disadvantages; 

b .  considered  j  oint  operational  plans ; 

c.  examined  sufficient  feasible  alternatives; 

d.  discussed  key  assumptions  and  variables  and  sensitivity  to  changes  in  these; 

e.  calculated  costs;  and 

f.  assessed  the  following: 

(1)  technology  risk  and  maturity; 

(2)  alternative  ways  to  improve  the  energy  efficiency  of  DoD  tactical  systems  with 
end  items  that  create  a  demand  for  energy,  consistent  with  mission  requirements  and  cost 
effectiveness;  and 

(3)  appropriate  system  training  to  ensure  that  effective  and  efficient  training  is 
provided  with  the  system  (DoDI  5000.02,  2008,  pp.  58-59). 

Weapon  Systems  Acquisition  Reform  Act  of  2009 

The  WSARA  of  2009  has  altered  the  AoA  process.  As  noted  in  the  previous  section, 
WSARA  (2009)  has  created  a  Director,  OSD  CAPE  and  an  organization  OSD  CAPE  to  develop 
guidance  for  AoAs  for  MDAPs  and  perfonn  this  analysis  as  directed  by  the  Secretary  of 
Defense.  OSD  CAPE  combined  the  responsibilities  of  the  previous  Director,  Program  Analysis 
&  Evaluation  (DPA&E),  and  the  Chairman,  Cost  Analysis  Improvement  Group  (CAIG)  (Brown, 
2009).  The  OSD  CAPE  office  also  must  “Review,  analysis,  and  evaluation  of  programs  for 
executing  approved  strategies  and  policies,  ensuring  that  infonnation  on  programs  is  presented 
accurately  and  completely,  and  assessing  the  effect  of  spending  by  the  DoD  on  the  U.S. 
economy”  (WSARA,  2009,  p.  4,  §  139c.).  OSD  CAPE  also  has  a  significant  role  for  independent 
cost  analysis  of  DoD  and  the  Services.  This  role  is  also  related  to  AoAs,  because  cost  estimation 
is  a  critical  component  of  AoAs. 
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The  AoA  guidance  must  consider  the  following  as  a  minimum: 

(1)  Full  consideration  of  possible  tradeoffs  among  cost,  schedule,  and  performance 
objectives  for  each  alternative  considered. 

(2)  An  assessment  of  whether  or  not  the  joint  military  requirement  can  be  met  in  a 
manner  that  is  consistent  with  the  cost  and  schedule  objectives  recommended  by  the 
Joint  Requirements  Oversight  Council  (JROC).  (WSARA,  2009,  p.  18,  Title  II 
Acquisition  Policy,  SEC.  201) 

WSARA  (2009,  p.  18,  Title  II  Acquisition  Policy,  SEC.  201)  also  alters  Section  2366a(a) 
of  Title  10,  U.S.C.  (2011)  by  adding  that  the  MDA  must  certify  that  “an  analysis  of  alternatives 
has  been  perfonned  consistent  with  study  guidance  developed  by  the  Director  OSD  CAPE.” 

DoD  Implementation  of  the  Weapon  Systems  Acquisition  Reform  Act  of  2009 

To  implement  WSARA  within  DoD  Components,  the  USD(AT&L)  issued  Directive- 
Type  Memorandum  (DTM)  09-027,  December  4,  2009.  DTM  09-027  (2009)  amended  the 
acquisition  policy  in  the  DoDI  5000.02  (2009),  the  Defense  Federal  Acquisition  Regulation 
(DFAR)  Supplement  (DFARS)  (1998),  and  the  Defense  Acquisition  Guidebook  (DAG).  The 
directive  states  that  “the  OSD  CAPE  leads  the  development  of  study  guidance  for  the  AoA  for 
each  joint  military  requirement  for  which  the  Chainnan  of  the  JROC  is  the  validation  authority” 
(DTM  09-027,  2009,  p.  3).  The  MDA  initiates  the  AoA  in  the  acquisition  decision  memorandum 
(ADM)  from  the  MDD  (i.e.,  the  start  of  the  MSA  phase),  and  provides  the  OSD  CAPE  study 
guidance  as  an  attachment  (DTM  09-027,  2009).  For  ACAT  ID  systems,  the  USD(AT&L)  is  the 
MDA.  The  policy  impact,  as  stated  by  DTM  09-027  (2009,  p.  3),  is  that  “The  MDA  directs 
initiation  of  the  AoA,  but  no  longer  approves  the  AoA  Study  Guidance,  which  is  now  approved 
by  the  OSD  CAPE.”  Also  for  an  MDAP,  the  MDA  must  certify  that  all  statutory  requirements 
have  been  completed  for  MS  A  and  MS  B,  including  the  conduct  of  an  AoA. 

On  September  13,  2010,  the  USD(AT&L)  issued  DTM  10-017 — Development  Planning 
to  Inform  MDD  Reviews  and  Support  AoA.  DTM  10-017  (2010)  states  the  following: 

“The  DoD  Components  shall  provide  evidence  at  the  MDD  Review  that  will  facilitate  the 
MDA’s  detennination  that: 

•  “The  candidate  materiel  solution  approaches  have  the  potential  to  effectively  address  the 

capability  gap(s),  desired  operational  attributes,  and  associated  dependencies. 
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•  “There  exists  a  range  of  technically  feasible  solutions  generated  from  across  the  entire 
solution  space,  as  demonstrated  through  early  prototypes,  models,  or  data. 

•  “Consideration  has  been  given  to  near-term  opportunities  to  provide  a  more  rapid  interim 
response  to  the  capability  need. 

•  “The  plan  to  staff  and  fund  analytic,  engineering,  and  programmatic  activities  supports 
the  proposed  MS  entry  requirements  as  identified  in  Reference  (c).”  (DTM  10-017,  2010, 

p.  2) 

DTM  10-017  (2010)  also  directs  that  the  Director,  Defense  Research  and  Engineering 
(DDR&E)  cooperate  with  the  Director,  OSD  CAPE,  and  be  a  participant  and  technical  adviser  in 
development  of  AoA  Study  Guidance  to  facilitate  consideration  of  technology  and  engineering 
risks  for  the  alternatives. 

The  Center  for  Strategic  and  International  Studies  developed  an  independent  review  of 
the  implementation  of  WSARA  and  noted  the  following  regarding  OSD  CAPE: 

•  Director,  OSD  CAPE  position,  was  filled  with  Christine  Fox  on  November  3,  2009. 

•  OSD  CAPE  leads  and  approves  AoA  study  guidance. 

•  OSD  CAPE  provides  policies  and  procedures  for  all  DoD  cost  estimates. 

•  OSD  CAPE  reviews  DoD  Component  cost  estimates  and  analyses  conducted  for  MDAPs 
and  Major  Automated  Infonnation  Systems  (MAIS). 

•  OSD  CAPE  conducts  Independent  Cost  Estimate  (ICE)  and  cost  analyses  for  MDAPs  and 
MAIS  for  which  USD(AT&L)  is  the  MDA  (Berteau,  Hofbauer,  &  Stephanie,  2010). 

The  important  aspects  in  the  DTMs  related  to  AoA  are  the  following:  OSD  CAPE  has 
been  created,  OSD  CAPE  develops  the  guidance  for  MDAP  AoAs,  OSD  CAPE  is  supported  by 
DDR&E,  technology  and  engineering  risk  are  stressed,  and  that  the  MDA  has  to  certify  that  an 
AoA  was  conducted. 

U.S.  Army  Initiatives  to  Improve  AoAs 

There  have  been  studies,  articles,  and  white  papers  written  about  how  to  improve  Anny 
AoAs  in  response  to  WSARA.  The  Office  of  the  Deputy  Assistant  Secretary  of  the  Anny  for 
Cost  and  Economics  (DASA-CE)  performed  a  study  to  examine  risk  considerations  in  pre-MS  A 
cost  analysis,  which  would  be  used  in  AoAs.  The  summary  and  conclusions  of  this  study  are  as 
follows: 

•  Pre-MS  A  estimates  must  be  accompanied  by  risk/uncertainty  ranges. 
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•  Risk  analysis  pre-MS  A  should  be  assessed  at  a  less  granular  level  (i.e.,  development 
schedule  risk,  instead  of  schedule  risk  of  development  science,  technology,  and 
engineering  [ST&E]). 

•  Typical  risk  analysis  approaches  can  be  tailored  to  pre-MS  A,  but  highly  detailed 
methods  are  rarely  useful. 

•  Schedule  risk  analysis  during  the  pre-MS  A  time  frame  can  be  particularly  useful. 

•  “Over-specification”  should  be  avoided  and  balance  achieved  between  the  competing 
demands  of  reasonable  assumption-making  and  development  of  a  “Cost  Analysis 
Requirements  Description  (CARD)-like”  level  of  detail. 

•  Methodologies  and  assumptions  must  be  consistent  among  alternatives  within  an  AoA  in 
order  to  ensure  maximum  comparability  (Roper,  2010). 

Both  the  risk  analysis  (schedule  and  cost)  and  consistent  methodology  and  assumptions 
for  alternatives  are  important  aspects  of  AoAs. 

The  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center  (TRAC)  has 
developed  white  papers  associated  with  AoA  issues  and  recommendations.  One  of  these  white 
papers  states  that  “AoA  policy  and  guidance  need  to  be  readdressed  to  reconcile  whether,  what 
and  why  the  AoA  should  report  the  same  or  very  similar  information  that  the  PM  [program 
manager]  is  obligated  to  report  for  the  program”  (TRAC,  2011a).  This  white  paper  addressed  two 
problems  associated  with  AoAs  to  include  the  following: 

•  The  purpose  and  scope  of  the  AoA  are  not  clearly  described  and  differentiated  for  each 
MS  decision. 

•  The  term,  “materiel  solution”  is  used  repeatedly  throughout  existing  AoA  guidance,  in 
some  instances  interchangeably  with  “alternative,”  without  expressing  a  clear  definition 
of  what  it  represents  and  using  language  that  suggests  an  inconsistently  applied 
interpretation  of  “materiel  solution”  (TRAC,  2011a). 

The  white  papers  provide  ideas  for  AoA  purpose  and  scope  for  both  MS  A  and  MS  B  and 
suggest  that  AoA  policy  provides  the  specific  decision  issues.  Suggestions  are  provided  in  the 
white  paper  for  a  distinction  between  the  terms  “materiel  solutions”  and  “alternatives”  in  the 
context  of  AoAs. 

Other  Army  papers  and  presentations  outline  improvements  to  analyses  and  AoA 
processes.  Various  presentations  outline  the  need  for  analyses  pre-MDD  and  to  support  the 
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requirements  developmental  process.  Hume  (2012)  presents  ideas  for  building  an  analytical 
framework  for  the  capabilities  development  process.  TRAC  (2011b)  presents  recommendations 
to  improve  the  Capabilities  Based  Assessment  (CBA)  process  and  to  introduce  new  analysis 
prior  to  MDD.  Other  presentations  provide  recommendations  associated  with  AoAs.  The  Army 
Capability  Integration  Center  (ARCIC),  TRAC,  and  AMSAA  (2011)  reviewed  lessons  learned 
from  recently  completed  AoAs  and  provided  recommendations  on  improving  AoAs. 
Recommendations  are  related  to  the  following:  refining  the  purpose  and  scope  of  MS  A  and  MS 
B  AoAs,  obtaining  industry  input  for  the  MS  A  AoA,  and  assigning  central  responsibility  for 
obtaining  foreign  system  data. 

OSD  CAPE  Suggestions  for  AoAs 

OSD  CAPE  developed  a  presentation  of  the  statute,  policy,  and  recent  practices 
associated  with  AoAs.  The  presentation  outlines  what  an  ideal  AoA  would  provide  and  states 
that  “AoA  provides  well-described  and  understood  “options”  for  the  decision-maker”  (Duke  & 
Kaspersen,  2011,  p.  8).  Duke  and  Kaspersen  (2011,  p.  8)  lists  the  elements  of  an  ideal  AoA  as 
the  following: 

•  Prioritized,  quantified,  contextualized  capability  gaps. 

•  Feasible  alternatives  that  are  affordable  and  mitigate  the  capability  gaps. 

•  The  trades  within  and  across  alternatives  among  cost,  schedule,  and  performance. 

•  Transparent  analysis  of  recognized  quality. 

•  Sensitivity  analysis:  How  robust  are  the  solutions? 

Duke  and  Kaspersen  (2011,  p.  9)  also  list  the  enablers  for  an  ideal  AoA,  which  include 
“effective  communications,  trust,  and  shared  vision  of  supporting  the  decision-maker  must 
underpin  all  efforts.”  Also  noted  in  the  report  is  that  a  clear  direction  for  the  AoA  is  needed  and 
any  changes  in  direction  should  be  relayed  quickly  to  the  Services.  Duke  and  Kaspersen  (2011, 
p.  9)  also  note  that  OSD  CAPE  must  have  “timely  insight  into  on  going  work,  supporting  tools, 
and  emerging  results — enables  OSD  to  understand  results.”  Duke  and  Kaspersen  note  good 
practices  for  AoAs,  which  include  the  following: 

•  Service  Executive  Steering  Body  is  formed  and  includes  key  stakeholders  (e.g., 
requirements,  acquisition,  and  resources). 

•  Sensitivity  analysis  performed  to  determine  the  dependence  of  results  on  key  parameters. 
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•  Cost  and  affordability  boundaries  established  early  in  the  analysis  to  screen  out 
unaffordable  candidates. 

•  OSD  CAPE  site  visits  to  centers  of  analysis  and  open  analyst  interaction  with  OSD 
CAPE. 

•  Contractor  information  is  well-protected  to  Source  Selection  Evaluation  Board  problems. 
Husband  and  Kaspersen  discuss  the  role  of  AoAs  and  OSD  CAPE  under  WSARA  of 

2009  and  provide  recommendations  on  program  cost  and  affordability  analyses,  interactions 
between  OSD  CAPE  and  AoA  team,  and  interactions  between  requirements,  acquisition,  and 
resources  communities.  Husband  and  Kaspersen  (2012,  p.  1)  define  an  AoA  as  “a  structured, 
unbiased  framework  for  systematically  evaluating  the  most  promising  alternatives  for  a  given 
need,  comparing  the  relative  merits  and  risks  of  those  alternatives,  and  providing  a  recommended 
approach  to  decision-makers.”  They  also  describe  the  purpose  of  an  AoA  is  “to  fully  evaluate 
and  fairly  compare  the  effectiveness  and  cost  of  all  the  viable  alternatives”  (Husband  & 
Kaspersen,  2012,  p.  4).  Husband  and  Kaspersen  discuss  how  cost  and  affordability  should  be 
considered  earlier  in  the  acquisition  process.  They  note  the  challenge  of  performing  a  cost 
estimate  at  MS  A,  because  the  systems  are  not  well-defined  at  that  point.  It  is  also  noted  that, 
“Senior  leaders  now  expect  the  Services  to  be  able  to  articulate  how  and  why  their  recommended 
alternative  is  affordable,  especially  when  the  recommended  option  has  a  higher  life  cycle  cost 
than  lower  performing  alternatives”  (Husband  &  Kaspersen,  2012,  p.  3).  Husband  and  Kaspersen 
also  recommend  that  OSD  CAPE  and  the  Anny  AoA  team  work  closer  together,  and  the 
requirements,  acquisition,  and  resourcing  teams  work  closer  together.  “The  AoA,  more  than  any 
other  acquisition  document,  is  where  the  requirements,  acquisition  and  resourcing  communities 
come  together  to  form  a  common  view  of  which  affordable  alternative  is  the  most  desirable  to 
pursue”  (Husband  &  Kaspersen,  2012,  p.  5). 
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CHAPTER  3 

RESEARCH  METHODOLOGY 

Selected  personnel  were  interviewed  for  this  research,  including  senior-level  Anny 
leadership  involved  with  AoAs  and  senior-level  OSD  CAPE  personnel.  Senior-level  personnel 
are  defined  as  SES  personnel,  COL  (or  0-6  equivalents),  and  GS-15  (or  equivalents).  These 
senior-  level  personnel  have  been  involved  with  U.S.  Army  AoAs  since  passage  of  WSARA  of 
2009.  Their  involvement  may  have  been  developing  guidance,  reviewing,  approving,  or 
certifying  U.S.  Anny  AoAs.  Members  of  the  senior-level  Army  leadership  involved  with  AoAs 
have  a  broad  view  of  all  the  AoAs  completed  and  sent  to  OSD.  Their  suggestions  on  how  to 
improve  the  AoAs  were  drawn  from  their  broad  perspective.  The  OSD  CAPE  interviews  yielded 
the  customer  perspective. 

The  interviews  were  completed  in  person  or  by  telephone.  The  interview  questions  were 
sent  beforehand  to  the  interviewees  so  they  could  prepare  for  the  interview.  After  the  interview, 
the  responses  were  sent  back  to  the  interviewee  to  ensure  the  conect  meaning  was  captured. 

The  results  of  the  interviews  were  divided  into  three  areas:  Army  personnel  associated 
with  organizations  conducting  AoAs  (i.e.,  the  AoA  team  organizations);  Anny  personnel 
associated  with  organizations  providing  oversight  to  AoAs;  and  OSD  CAPE  personnel 
associated  with  Army  AoAs.  The  findings  are  divided  into  these  three  areas  to  compare  U.S. 
Anny  and  OSD  CAPE  responses  to  the  interview  questions. 

Army  AoA  Team  Interviewees 

The  section  identifies  the  U.S.  Anny  AoA  Team  interviewees,  which  include  personnel 
leading  organizations  that  perfonn  work  on  AoAs.  Interviewees  from  the  following  organizations 
were  interviewed:  TRAC,  AMSAA,  and  the  Office  of  DASA-CE.  These  organizations  fonn  the 
core  team  that  performs  AoAs.  Other  organizations  work  on  AoAs,  including  the  Research 
Development  and  Engineering  Command  (RDECOM),  Army  Research  Laboratory  (ARL),  and 
Research,  Development  and  Engineering  Centers  (RDECs).  Many  other  organizations  support 
AoAs,  including  Program  Executive  Officers  (PEOs),  PMs,  TRADOC  Centers  of  Excellence 
(COEs),  the  Anny  Staff,  and  the  Assistant  Secretary  of  the  Anny  for  Acquisition,  Logistics  and 
Technology  (ASA(ALT)).  The  interviews  were  limited  to  TRAC,  AMSAA,  and  the  office  of 
DASA-CE  because  of  time  constraints. 
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TRAC  typically  is  the  Army  lead  for  AoAs,  and  it  performs  the  following  analyses: 
force-on-force  modeling;  combat  effectiveness;  operational  effectiveness;  sustainment;  Doctrine, 
Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel  and  Facilities 
(DOTMLPF);  gap;  transportability;  and  a  portion  of  cost.  AMSAA  is  a  member  of  the  AoA 
team,  and  it  perfonns  the  following  analyses:  perfonnance,  including  survivability,  force 
protection,  lethality,  mobility,  fuel  consumption,  sensors,  electronic  warfare,  communications, 
network,  and  chemical/biological;  logistics,  including  reliability,  Class  IX  spares,  and  fuel 
consumption;  growth;  risk  assessment,  including  schedule  and  technical;  and  trades,  including 
cost,  schedule,  and  technology.  The  Office  of  the  DASA-CE  is  a  member  of  the  AoA  team,  and 
it  performs  cost  analysis,  cost  risk  analyses,  and  trades  analyses. 

The  population  of  interviewees  includes  SESs  that  are  most  closely  involved  with  Army 
AoAs  and  select  COL  and  GS-15  level  personnel  involved  in  AoAs.  The  COL  and  GS-15 
personnel  were  selected  because  they  have  been,  and  continue  to  be,  involved  in  multiple  U.S. 
Anny  AoAs.  The  name,  rank,  organization,  and  position  are  provided  below  for  each  person 
interviewed. 

Dr.  William  Forrest  Crain,  Director,  AMSAA — (SES) 

Mr.  Michael  Bauman,  Director,  TRAC — (SES) 

Mr.  Patrick  O’Neill,  Technical  Director,  AMSAA — (SES) 

Ms.  Pamela  Blechinger,  Director,  TRAC  Fort  Leavenworth  (FLVN) — (SES) 

Dr.  Garry  Lambert,  Director,  TRAC  White  Sands  Missile  Range  (WSMR) — (SES) 

Mr.  Stephen  Barth,  DASA-CE— (SES) 

COL  Douglas  Hersh,  Deputy  Director,  TRAC  WSMR 

COL  Bradley  Pippin,  Deputy  Director,  TRAC  FLVN 

Mr.  Robert  Chandler,  Chief,  Weapon  System  Analysis  Division,  AMSAA 

Mr.  Frank  Decker,  TRAC  FLVN 

Mr.  Rod  Eaton,  TRAC  WSMR 

Mr.  Scott  Schoeb,  Chief,  Maneuver  Systems  Branch,  AMSAA 
Mr.  Alan  Dinsmore,  Integration  Team  Leader,  AMSAA 
Mr.  David  Henningsen,  Office  of  the  DASA-CE 
Mr.  Morteza  Anvari,  Office  of  the  DASA-CE 
Mr.  Sean  Vessey,  Office  of  the  DASA-CE 
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Ms.  Alison  Tichenor,  Office  of  the  DASA-CE 
Army  AoA  Oversight  Organizations  Interviewees 

The  section  identities  the  interviewees  from  U.S.  Anny  AoA  oversight  organizations, 
which  include  the  Military  Operations-Capabilities  Integration  (DAMO-CI),  Deputy  Chief  of 
Staff  (DCS),  G-3/5/7  and  ARCIC.  These  organizations  provide  oversight  to  Army  AoAs. 
DAMO-CI,  DCS  G-3/5/7  provides  draft  AoA  guidance  to  OSD  CAPE,  directs  Anny 
organizations  to  perform  the  AoAs,  and  reviews  the  AoAs.  ARCIC  directs  Anny  organizations 
to  perform  the  AoA  and  reviews  the  AoAs. 

The  population  of  interviewees  includes  SESs  that  are  most  closely  involved  with  Anny 
AoAs  and  select  colonel  and  GS-15  level  personnel  involved  in  AoAs.  The  colonel  and  GS-15 
personnel  were  selected,  because  they  have  been,  and  continue  to  be,  involved  in  multiple  Army 
AoAs.  The  name,  rank,  organization,  and  position  are  provided  below  for  each  person 
interviewed. 

Dr.  David  Markowitz,  Director,  DAMO-CI  DCS  G-3/5/7 — (SES) 

Mr.  Allan  Resnick,  Director  of  Assessment,  Architecture  &  Mission  Command 
Directorate,  ARCIC — (SES) 

COL  Robert  Hume,  Chief,  Studies  and  Analysis,  ARCIC 

COL  Richard  Holdren,  Division  Chief,  Operations  Research/Systems  Analysis,  DAMO- 
CI  DCS  G-3/5/7 

Mr.  Michael  Moore,  Deputy  Director,  DAMO-CI  Prioritization  and  Analysis,  DCS  G- 
3/5/7 

OSD  CAPE  Interviewees 

This  section  identifies  the  OSD  CAPE  interviewees.  The  following  personnel  were 
interviewed  from  OSD  CAPE  to  gain  the  AoA  customer  perspective.  Each  of  these  interviewees 
has  been  involved  with  Anny  AoAs.  The  majority  of  Anny  AoAs  are  coordinated  through  the 
OSD  CAPE  Land  Forces  Division,  which  reports  to  the  OSD  CAPE  Deputy  Director  for 
Program  Evaluation. 

Dr.  Scott  Comes,  OSD  CAPE  Deputy  Director  for  Program  Evaluation  (SES) 

Ms.  Elaine  Simmons,  Director,  Land  Forces  Division  (SES) 

Mr.  Keith  Kaspersen,  Land  Forces  Division 

Mr.  John  Duke,  Land  Forces  Division 
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Dr.  Eric  Fulmer,  Land  Forces  Division 

Mr.  Chuck  Allen,  Command,  Control,  Communications,  and  Computers  (C4)  and 
Infonnation  Programs  Division 

Interview  Questions 

Each  person  interviewed  for  this  research  was  asked  the  following  questions: 

1 .  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

2.  What  do  you  think  are  the  major  issues  associated  with  AoAs  under  the  new  WSARA 
guidance? 

3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

4.  What  types  of  analyses  do  you  think  need  to  be  perfonned  in  AoAs  to  meet  OSD 
CAPE  expectations? 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  perfonnance  should  be 
considered  in  AoAs? 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE 
expectations? 

The  interview  answers  were  written  and  submitted  back  to  the  interviewee  for  any 
modifications,  additions,  or  deletions. 
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CHAPTER  4 
INTERVIEW  RESULTS 

This  chapter  provides  the  results  of  the  Army  and  OSD  CAPE  interviews.  A  summary  is 
provided  of  interview  answers  for  each  of  the  eight  questions.  The  Anny  AoA  team,  Anny  AoA 
oversight  organizations,  and  OSD  CAPE  answers  are  provided  separately  so  they  can  be 
compared. 

Summary  of  Army  AoA  Team  Interviews 

This  section  provides  a  summary  of  the  Army  AoA  team  interviewees.  A  summary  of  the 
interview’s  key  points  are  provided  for  each  question  as  bullet  points.  Appendix  A,  Anny  AoA 
Team  Organizations  Interview  Results,  expands  on  these  key  points  by  providing  the 
interviewees’  specific  points. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

•  The  purpose  of  AoAs  is  to  frame  the  decision  space  and  to  identify  the  best  investment 
strategy  and  acquisition  option.  There  is  no  best  alternative,  but  rather  a  best  acquisition 
decision. 

•  The  purpose  of  AoAs  is  to  inform  the  decision-makers  on  best  investment  courses  of 
action. 

•  The  purpose  of  an  AoA  is  to  identify  the  most  affordable,  cost-effective  solution  through 
tradeoff  and  cost-benefit  analyses. 

•  The  purpose  of  AoAs  is  to  inform  decision-makers  on  benefits  of  materiel  solutions  to 
meet  capability  gaps. 

•  AoAs  are  used  to  determine  required  performance  levels  and  desired  attributes  through 
cost,  schedule,  and  performance  trades. 

•  AoAs  are  comprised  of  cost,  schedule,  and  perfonnance  analyses  on  potential  materiel 
solutions. 

•  AoAs  should  support  the  acquisition  community  by  identifying  the  cost,  schedule,  and 
perfonnance  risks. 

•  AoAs  should  be  unbiased  with  no  preferred  solution,  informed  about  the  funding 
available,  and  aligned  to  national  security,  defense,  and  military  strategies. 
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•  Analyses  should  be  perfonned  before  the  MS  A  AoA  to  identify  capability  gaps.  The 
AoA  should  start  as  soon  as  a  materiel  solution  is  identified. 

2.  What  do  you  think  the  major  issues  are  associated  with  AoAs  under  the  new  WSARA 
guidance? 

•  WSARA  has  had  a  positive  impact  on  AoAs  by  increasing  the  importance  of  analysis  in 
MS  decisions.  Under  WSARA,  AoAs  have  to  be  more  accurate,  detailed,  and  unbiased. 
AoAs  can  better  support  the  requirements  development  process. 

•  WSARA  has  increased  OSD  CAPE’s  influence  in  the  AoA  process.  OSD  CAPE  provides 
the  AoA  guidance  to  the  Services  and  detennines  if  the  AoA  has  been  completed  per 
their  guidance.  WSARA  also  requires  that  the  DAE  [Defense  Acquisition  Executive] 
certifies  that  the  AoA  has  been  completed  in  accordance  with  the  OSD  CAPE  guidance. 
WSARA  also  provided  a  political  appointee  to  lead  OSD  CAPE. 

•  WSARA  has  also  caused  friction  between  different  organizations  in  OSD  including 
USD(AT&L),  OSD  CAPE,  Policy,  and  the  JS  (JS). 

•  Under  WSARA,  tradeoff  analysis  for  cost,  schedule,  and  perfonnance  are  required  across 
and  within  alternatives.  The  trades  are  at  the  system  and  subsystem  levels,  and  they  are 
conducted  throughout  the  AoA.  WSARA  has  dictated  cost,  schedule,  and  performance 
trades  across  and  within  the  alternatives. 

•  WSARA  requires  technical/perfonnance,  cost,  and  schedule  risk  analyses.  The  risk 
analysis  is  associated  with  the  cost,  schedule,  and  performance  tradeoff  analysis. 

WSARA  has  increased  the  emphasis  on  cost  and  schedule  analyses  in  AoAs. 

•  Under  WSARA,  the  scope  of  AoAs  has  increased.  Cost,  schedule,  and  perfonnance  must 
be  estimated,  traded,  and  assessed  for  risk.  The  tradeoff  analyses  and  OSD  CAPE 
guidance  have  increased  the  number  of  alternatives. 

•  Because  of  the  increased  complexity  and  interest  in  AoA  from  WSARA,  the  Anny  has 
experienced  an  increased  workload  in  the  analytical  community.  This  increased  workload 
requires  additional  analysts  (cost  and  performance)  and  funding  to  support  conducting 
AoAs. 

3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

•  AoAs  should  be  tailored  specifically  to  the  program,  the  decision  space,  and  the  different 
MS  decisions. 
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•  More  information  and  data  are  available  for  the  later  MS  AoAs  as  opposed  to  the  MS  A 
AoAs.  The  resolution  of  the  alternatives  is  greater,  and  there  is  more  knowledge  of  the 
perfonnance. 

•  As  a  system  progresses  through  MS  A,  MS  B,  and  MS  C  decisions,  the  program  (i.e., 
budgets  and  timelines)  becomes  more  concrete  and  constrained.  There  are  more  tradeoffs 
between  cost,  schedule,  and  perfonnance  at  MS  A. 

•  There  are  more  alternatives  and  courses  of  action  at  MS  A  than  MS  B.  MS  A  AoA  should 
consider  a  wide  range  of  technologies  and  systems. 

•  The  detail,  fidelity,  and  robustness  of  analyses  in  AoAs  increase  from  MS  A  to  MS  B  and 
MS  C.  More  detailed  scenarios  are  used  as  the  program  progresses. 

•  At  MS  A,  the  AoA  should  include  a  thorough  tradeoff  analysis  of  cost,  schedule,  and 
perfonnance.  The  MS  A  AoA  should  model  a  set  of  attributes  as  an  alternative,  and  the 
alternative  should  not  be  associated  with  a  system.  At  MS  A,  alternatives  can  serve  as 
vessels  to  hold  the  attributes  together  to  provide  a  better  understanding  of  the 
interdependencies  of  attributes.  Performance  and  cost  estimates  will  have  to  be  quickly 
made  for  the  evolving  alternatives.  First-order  estimates  of  perfonnance  and  cost  will 
have  to  be  made. 

•  The  tradeoff  analyses  continue  at  the  beginning  of  the  MS  B  AoA,  but  later  in  the  AoA 
the  alternatives  should  be  solidified.  The  MS  B  AoA  would  compare  the  alternatives  to 
the  requirement  and  to  each  other.  The  MS  B  AoA  should  consider  broader  acquisition 
options  to  fulfill  the  needs. 

•  Although  WSARA  does  not  require  a  new  AoA  for  each  MS  decision,  conducting  AoAs 
for  each  MS  decision  would  be  beneficial.  Having  formal  AoAs  at  the  later  MSs  would 
increase  the  fidelity  and  structure  of  the  analysis  and  would  increase  the  information  flow 
for  the  analyses. 

•  It  would  be  beneficial  to  obtain  input  from  industry  for  MS  A. 

•  AoA  analysts  must  have  better  understanding  of  their  customers,  stakeholders,  and 
external  operating  environment. 

4.  What  types  of  analysis  do  you  think  need  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 
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•  Tradeoff  analyses  between  cost,  schedule,  and  technology/performance  should  be 
perfonned  throughout  the  AoA.  The  trade  space  must  be  fully  explored. 

•  Risk  assessments  should  be  conducted  for  cost,  schedule,  and  technical. 

•  AoAs  must  address  cost  vs.  benefits  and  cost  vs.  performance/effectiveness. 

•  AoAs  should  consider  Operations  and  Support  (O&S)  cost  and  Life  Cycle  Cost  (LCC)  in 
addition  to  the  unit  costs. 

•  AoAs  should  consider  other  aspects  besides  effectiveness,  such  as  the  following: 
reliability,  DOTMLPF,  strategic  impacts,  transportability,  maintainability,  manpower 
analyses,  force  structure  impacts,  and  squad  size  impacts. 

•  AoAs  should  include  the  following  types  of  analyses:  mission  effectiveness, 
perfonnance,  cost,  affordability,  tradeoff  (cost,  schedule,  performance/technology), 
sensitivity/  parametric,  risk  (cost,  schedule,  and  technical),  growth  assessment, 
sustainment,  fuel  consumption,  energy  consumption,  DOTMLPF,  fleet  strategy,  and 
portfolio. 

•  The  analyses  must  be  transparent,  open,  and  understandable  to  OSD  CAPE.  The  analyses 
have  to  address  OSD  CAPE’s  decision  choices  and  “tipping  points.”  Analyses  have  to  be 
objective,  robust,  and  credible. 

•  OSD  CAPE  guidance  drives  the  types  of  analyses  perfonned  in  AoAs,  but  the  guidance  is 
not  consistent  across  AoAs. 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

•  There  are  four  basic  types  of  alternatives  to  include  the  system  in  service,  modifications 
of  the  system  in  service,  ND I/foreign  systems,  and  new  start  programs. 

•  A  screening  analysis  conducted  during  the  AoA  would  be  beneficial  in  detennining  an 
initial  set  of  alternatives.  The  screening  analyses  start  with  a  broad  look  for  possible 
alternatives  and  reduce  the  number  of  alternatives,  based  on  criteria. 

•  OSD  CAPE  plays  a  significant  role  in  the  development  of  alternatives.  OSD  CAPE 
provides  the  AoA  guidance  and  chairs  the  Senior  Advisory  Group  (SAG),  which 
approves  the  alternatives.  OSD  CAPE  should  not  be  involved  with  the  Services’  force 
design. 

•  The  number  of  alternatives  should  be  managed  to  a  reasonable  number  (six  or  fewer)  to 
complete  the  AoA  in  the  9-  to  12-month  timeframe.  Alternatives  should  be  eliminated  if 
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they  are  determined  to  be  infeasible,  and  there  should  be  a  limited  number  of  “special 
interest”  systems. 

•  Defining  alternatives  is  both  a  top-down  and  bottoms-up  process.  The  top-down  process 
is  based  on  guidance  from  Anny  and  OSD  leadership,  and  the  bottoms-up  approach  is 
based  on  a  screening  analysis. 

•  For  MS  A,  the  alternatives  should  be  attribute-based  and  developed  by  the  materiel 
developer,  user,  analyst,  and  industry. 

•  Alternatives  could  be  a  mix  of  different  systems  for  the  fleet.  An  alternative  could 
include  some  new  start  systems  for  a  portion  of  fleet  and  some  modified  systems  for  the 
remainder  of  the  fleet. 

•  There  are  more  alternatives  in  an  MS  A  AoA. 

•  There  were  other  points  identified  for  the  process  of  defining  alternatives. 

6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

•  For  foreign  systems,  it  is  difficult  to  obtain  the  level  of  data  needed  for  AoA  performance 
and  cost  analyses.  Foreign  countries  or  companies  are  not  willing  to  share  detail  data  on 
their  systems,  because  they  don’t  believe  the  United  States  will  purchase  them.  At  some 
point,  data  might  be  available  for  one  configuration,  but  not  the  configuration  the  AoA 
requires. 

•  Foreign  systems  should  be  identified  and  agreed  upon  early  in  the  AoA  process.  The  data 
and  infonnation  request  process  should  start  early. 

•  The  AoA  team  should  not  have  the  responsibility  to  obtain  the  data  for  foreign  systems. 

•  An  Army  PM  is  required  to  develop  a  notional  schedule  for  the  foreign  alternative  as 
input  into  the  tradeoff  and  risk  analyses. 

•  All  foreign  systems  will  require  some  level  of  modification  to  address  U.S.  standards, 
U.S.  government-furnished  equipment  (GFE)  (e.g.,  radios),  or  U.S.  manufacturing  sites. 
Modifications  may  also  be  required  to  meet  desired  capability.  The  cost  and  schedule  of 
these  modifications  would  have  to  be  included  in  the  AoA. 

•  There  are  risks  associated  with  using  foreign  systems  such  as  compatibility, 
interoperability,  and  sustainment.  These  risks  must  be  considered  in  the  AoA. 

•  The  United  States  rarely  buys  foreign  systems  because  of  political  reasons  and  industrial 
base  reasons. 
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•  Foreign  alternatives  should  be  considered  appropriately  based  on  the  infonnation/data 
available.  There  may  be  greater  uncertainty  in  the  analytical  results  of  foreign  systems. 
This  uncertainty  should  be  understood  by  the  AoA  customers. 

•  The  number  of  foreign  alternatives  should  be  limited  (to  one  or  two),  because  of  the 
analytical  resources  to  evaluate  them. 

•  Foreign  systems  should  not  be  considered  if  they  are  not  viable,  feasible  candidates. 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs? 

•  The  trades  are  between  cost,  schedule,  and  technology.  These  trades  would  result  in  a 
level  of  perfonnance.  Tradeoff  analyses  involve  parametric  analyses  where  technologies 
are  added  or  removed  to  the  alternatives. 

•  A  systematic  approach  to  the  tradeoff  analysis  should  be  developed  and  used  for  AoAs. 

•  Affordability  analysis  should  be  conducted  to  determine  a  cost  target  range,  which  can  be 
used  in  the  tradeoff  analyses. 

•  The  tradeoffs  must  be  done  within  the  context  of  the  decision  space. 

•  Operational  effectiveness  and  performance  analyses  are  required  to  perfonn  the  tradeoff 
analyses. 

•  It  is  difficult  to  consider  schedule  trades  in  the  tradeoff  analysis.  The  desired  timeline  for 
the  development  of  the  system  may  preclude  changing  the  schedule.  There  is  a  lack  of 
realistic  schedule  estimation,  and  schedule  variations  greatly  affect  cost. 

•  The  tradeoff  analysis  should  be  done  before  an  AoA,  upfront  in  the  AoA  process  or 
throughout  the  AoA  process. 

•  The  tradeoff  analysis  will  become  difficult  with  multiple  alternatives  to  consider. 

•  Risk  assessments  are  part  of  the  tradeoff  analyses.  The  relationship  between  cost, 
schedule,  and  performance  must  be  understood. 

•  Risk  methodology  should  have  different  approaches  to  consider  past  systems/components 
and  new  system/components. 

•  Tradeoff  analyses  allow  more  robust,  cost-effective  solutions  to  be  developed. 

•  The  tradeoff  analyses  affect  the  requirements  generation  process.  Tradeoffs  can  be  used 
to  detennine  the  minimum  requirements  to  reduce  cost  and  maintain  schedule.  The  Joint 
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Capabilities  Integration  and  Development  System  (JCIDS)  process  should  also  consider 
trades  before  an  AoA  is  conducted. 

•  Currently,  cost  is  the  most  important  factor  in  the  decision  space. 

•  For  the  tradeoff  analysis,  LCC  should  be  considered  rather  than  only  procurement  costs. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

•  The  Anny  should  involve  OSD  CAPE  early  in  the  AoA  process.  The  Anny  must 
understand  OSD  CAPE’s  guidance,  and  OSD  CAPE  must  understand  the  Army’s 
proposed  study  plan.  The  Army  should  be  proactive  in  writing  draft  AoA  guidance  for 
OSD  CAPE’s  consideration. 

•  The  Army  must  increase  the  open  and  transparent  communication  with  OSD  CAPE 
during  the  AoA.  The  Anny  must  share  information  and  explain  the  analyses  to  OSD 
CAPE  often  during  the  AoA  process.  The  Anny  must  work  closely  with  OSD  CAPE  to 
increase  their  level  of  understanding  of  the  analysis  so  they  understand  that  the  analysis  is 
credible  and  relevant. 

•  The  Anny  must  ensure  the  AoA  results  are  presented  in  an  unbiased  manner.  The  Army 
should  involve  OSD  CAPE  in  the  interpretation  of  results.  There  should  be  no  preferred 
alternatives  in  AoAs. 

•  AoA  analyses  and  results  must  be  related  to  OSD  CAPE  decision  choices.  The  decision 
choices  are  not  necessarily  alternatives.  AoAs  must  inform  and  enable  decisions  and  be 
focused  on  the  decision  space. 

•  There  should  be  an  agreed-upon  set  of  alternatives  and  a  hierarchy  established  within  this 
set  to  ensure  analyses  is  performed  on  them. 

•  The  scope  of  an  AoA  should  be  reduced  as  it  progresses.  The  number  of  alternatives 
should  be  reduced  by  the  SAG  if  the  alternatives  are  no  longer  feasible  or  do  not  add  to 
the  decision  choices.  Analyses  that  do  not  add  information  relevant  to  the  decision  should 
be  stopped. 

•  AoAs  must  explore  trade  space  for  cost,  schedule,  and  performance.  Risk  assessment 
should  be  conducted  for  cost,  schedule,  and  technical  (including  integration). 

•  A  coherent,  robust  risk  and  trade  methodology  must  be  developed  by  the  Army.  AMSAA 
is  currently  working  on  this  approach,  and  DASA-CE  has  a  tool  to  support  trades. 
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•  Cost  methodology  should  be  refined  to  develop  more  estimating  relationships  that  link 
perfonnance  to  cost,  incorporate  risk  better,  and  consider  affordability. 

•  Conducting  fonnal  AoAs  for  each  MS  decision  would  increase  the  fidelity  and  structure 
of  the  analysis  and  would  increase  the  infonnation  flow  for  the  analyses. 

•  The  MS  A  AoA  results  should  be  used  to  develop  the  Capability  Development  Document 
(CDD). 

•  There  are  funding  challenges  associated  with  the  conduct  of  AoAs. 

•  There  should  be  realistic  schedules  for  conducting  AoAs. 

•  The  Anny  needs  better  unity  of  purpose. 

•  The  Army  should  examine  Navy  and  Air  Force  AoAs  that  OSD  CAPE  believes  are  well 
done. 

•  Other  recommendations  were  made  to  improve  AoAs. 

Summary  of  Army  AoA  Oversight  Organizations  Interviews 

This  section  provides  a  summary  of  the  Army  AoA  oversight  organizations  interviewees. 
A  summary  of  the  interview  key  points  is  provided  for  each  question  as  bullet  points.  Appendix 
B,  Army  AoA  Oversight  Organizations  Interview  Results,  expands  on  these  key  points  by 
providing  the  interviewees’  specific  points. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

•  AoAs  should  support  the  requirements  development  process.  AoAs  also  should  support 
multiple  customers  in  the  requirements  development  and  acquisition  communities. 

•  OSD  CAPE  is  focused  on  supporting  MS  decisions  with  AoAs  rather  than  supporting 
requirements  development  or  acquisition  processes. 

•  AoAs  should  inform  decisions  regarding  acquisition  (including  cost  and  schedule), 
requirements,  and  resource  allocation. 

•  Formal  AoAs  at  the  later  MS  decisions  are  needed  to  support  all  the  decision  points 
throughout  the  acquisition  process.  There  are  too  many  unknowns  early  in  the  program 
for  an  MS  A  AoA  to  answer  all  the  questions. 

•  The  purpose  of  an  AoA  is  to  develop  the  supporting  evidence  to  identify  an  achievable, 
affordable,  and  operationally  relevant  set  of  attributes.  Alternatives  are  the  “stalking 
horses”  to  better  understand  the  set  of  attributes.  Alternatives  are  the  surrogate  system 
that  allows  analysis  to  be  conducted  on  a  set  of  attributes. 
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2.  What  do  you  think  the  major  issues  are  associated  with  AoAs  under  the  new  WSARA 
guidance? 

•  Completing  an  AoA  at  MS  A  is  too  early,  but  completing  a  detailed  AoA  at  MS  B  is  too 
late.  The  correct  timing  for  an  AoA  is  at  the  Preliminary  Design  Review  (PDR). 

•  AoAs  should  not  try  to  answer  questions  that  are  already  being  answered  in  the 
acquisition  process  (e.g.,  source  selection). 

•  The  requirements  and  acquisition  communities  are  also  customers  of  AoAs  in  addition  to 
OSD  CAPE.  AoA  should  benefit  all  stakeholders  and  not  only  focus  on  OSD  CAPE. 

•  Capability  gaps  and  Concept  of  Operations  (CONOPS)  should  be  developed  before  the 
start  of  an  AoA. 

•  Timely  OSD  guidance  and  adequate  AoA  study  plans  are  needed. 

•  WSARA  gave  OSD  CAPE  more  influence  on  AoAs,  because  OSD  CAPE  has  to  declare 
an  AoA  sufficient  to  support  MS  decisions. 

•  WSARA  specifies  that  the  AoA  should  cover  cost,  schedule,  and  performance  tradeoffs. 
Requirements  can  be  traded. 

•  Anny  AoAs  have  increased  emphasis  focus  on  measurement  space  and  cost-benefit 
analyses. 

•  WSARA  has  increased  the  workloads  of  the  analytical  community,  and  the  Army  may 
not  have  the  resources  (funding  or  personnel)  to  perfonn  detailed  AoAs.  A  central 
funding  source  should  be  available  for  AoAs. 

•  The  Anny  does  not  have  enough  analysts  skilled  in  technical  risk  assessment,  cost 
analyses,  or  affordability  assessments. 

3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

•  AoA  must  be  tailored  to  the  specific  program,  requirements,  and  problems. 

•  The  timing  of  AoAs  is  not  well  coordinated  with  the  acquisition  process.  Cost,  schedule, 
and  performance  trades  at  MS  A  are  too  early  and  at  MS  B  are  too  late;  the  PDR  would 
be  better  timing  for  an  AoA. 

•  It  is  not  practical  to  have  one  AoA  to  support  all  MS  decisions. 

•  There  should  not  be  an  AoA  associated  with  every  MS. 

•  Before  MDD,  pre-AoA  analyses  must  identify  the  capability  gaps,  threats,  problems, 
attribute  trade  space,  and  feasible  alternatives. 
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•  For  MS  A,  the  AoA  should  inform  the  requirements  by  evaluating  the  cost,  schedule, 
perfonnance,  and  risk  of  the  concepts;  detennining  a  set  of  achievable,  affordable,  and 
operationally  relevant  attributes  for  the  system;  and  identifying  affordable,  operationally 
acceptable  solutions. 

•  For  MS  B,  the  AoA  should  identify  cost  drivers,  conduct  cost-performance  trades, 
support  CDD  development,  and  support  the  PDR.  The  AoA  should  also  answer  the 
question  of  whether  or  not  to  build  the  system. 

•  At  MS  C,  the  AoA  support  provides  infonnation  to  support  a  fielding  decision  for  the 
system  and  address  affordable  distribution  of  the  system. 

•  AoAs  should  identify  the  cost  of  capabilities  and  how  much  the  capabilities  can  be 
reduced  to  make  the  system  more  affordable. 

4.  What  types  of  analysis  do  you  think  needs  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 

•  Gap  analyses  must  be  conducted  to  provide  a  basis  for  the  AoA.  The  AoA  must  assess 
various  ways  to  fill  the  gap. 

•  Comprehensive  cost,  schedule,  and  performance  tradeoff  analyses  should  be  conducted  in 
an  AoA.  Risk  analyses  should  also  be  conducted.  Risk  estimates  can  be  associated  with 
the  given  cost,  schedule,  and  performance,  or  it  can  be  associated  with  the  trades. 

•  AoAs  should  concentrate  on  determining  a  set  of  achievable,  affordable,  and 
operationally  relevant  attributes  that  mitigate  an  operational  problem. 

•  AoAs  should  develop  cost  estimates  including  Average  Procurement  Unit  Cost  (APUC), 
Average  Manufacturing  Unit  Cost  (AMUC)  and  LCC. 

•  AoAs  should  not  address  the  acquisition  strategy,  the  number  of  contractors,  how  to  build 
the  system,  or  the  Technology  Demonstration  feeds  into  engineering  design. 

•  It  is  more  difficult  for  the  Army  to  meet  OSD  CAPE  expectations  than  the  Air  Force  or 
Navy.  The  Air  Force  and  Navy’s  materiel  solutions  are  more  closely  related  to  the 
strategic  level  than  the  Army’s.  OSD  CAPE  is  more  focused  at  the  strategic  level. 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

•  The  definition  of  the  alternatives  should  be  tailored  to  the  capabilities  of  the  system. 

•  Analyses  should  be  conducted  before  MDD  to  determine  alternatives. 

•  Screening  analysis  is  beneficial  to  detennine  the  alternatives. 
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•  Cost,  schedule,  and  performance  trades  should  be  conducted  within  the  alternatives. 

•  Parametric  analysis  should  be  perfonned  to  detennine  the  most  important  attributes. 

•  AoAs  should  compare  cases  rather  than  alternatives.  Cases  are  examples  of  alternatives. 

•  Alternatives  represent  “stalking  horses”  to  identify  an  achievable,  affordable,  and 
operationally  relevant  set  of  attributes.  The  purpose  of  having  alternatives  is  to  identify 
these  attributes. 

•  The  Anny  should  work  closely  with  OSD  CAPE  to  obtain  agreement  on  the  alternatives. 

6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

•  Foreign  systems  should  be  considered  as  equivalent  to  U.S.  systems  with  the  same 
fidelity  of  data. 

•  Foreign  systems  represent  NDI  cases — therefore,  they  should  not  be  modified. 

•  For  statutory  reasons,  the  United  States  may  not  purchase  a  foreign. 

•  Foreign  systems  should  only  be  considered  in  AoAs  for  comparison  purposes  only  to 
investigate  capabilities  and  attributes. 

•  Foreign  systems  will  have  to  be  modified  to  meet  U.S.  standards. 

•  Foreign  systems  should  be  discussed  with  OSD  CAPE  early  in  the  AoA  process  to 
detennine  how  they  will  be  considered. 

•  A  “concept  vehicle”  approach  to  developing  systems  could  enhance  AoAs. 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs? 

•  Tradeoff  analyses  should  be  tailored  to  the  capability,  maturity,  and  cost  of  the  system. 

•  Trades  should  be  performed  within  and  among  alternatives  to  address  the  decision  space. 

•  The  tradeoff  analysis  will  support  the  requirements  development  process,  including  the 
needed  attributes. 

•  Risk  assessment  should  be  performed  on  tradeoffs  between  cost,  schedule,  and 
perfonnance. 

•  There  were  differing  priority  of  factors  including  cost,  schedule,  and  perfonnance  and 
perfonnance,  cost,  and  schedule. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

•  AoAs  must  address  the  decision  process  and  provide  utility  to  the  decision-makers. 
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•  AoAs  must  satisfy  all  the  customers  including  OSD  CAPE,  the  acquisition  community, 
the  requirements  community,  and  the  portfolio  mix  community. 

•  AoAs  must  address  OSD  CAPE  guidance  and  issues. 

•  The  Anny  must  engage  OSD  CAPE  earlier  in  the  AoA  process.  The  Army  must  be 
candid  and  open  with  OSD  CAPE. 

•  The  AoA  guidance  should  only  include  three  points:  (1)  the  purpose  for  the  AoA;  (2) 
when  the  AoA  results  are  needed;  and  (3)  identify  the  key  angst/friction/concern  points  or 
issues. 

•  Anny  requirements  must  be  developed  with  the  support  of  analyses. 

•  The  Anny  needs  more  analytic  capability  to  address  AoAs. 

•  The  Army  could  utilize  a  “concept  vehicle  show”  for  industry  to  demonstrate  their 
technologies. 

Summary  of  OSD  CAPE  Interviews 

This  section  provides  a  summary  of  the  OSD  CAPE  interviewees.  A  summary  of  the 
interview  key  points  is  provided  for  each  question  in  bullet  points.  Appendix  C,  OSD  CAPE 
Interview  Results,  expands  on  these  key  points  by  providing  the  interviewees’  specific  points. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

•  There  are  numerous  purposes  to  an  AoA,  including  providing  information  to  inform 
acquisition  decisions,  exploring  the  trade  space  of  cost  schedule  and  perfonnance,  and 
answering  the  “Why”  questions. 

•  There  are  desired  characteristics  of  the  options  and  alternatives  to  include:  objectivity, 
nonpreferential  options,  affordable  alternatives,  and  a  robust  set  of  alternatives. 

•  The  analyses  should  back  up  a  recommended  solution,  show  due  diligence,  explain  the 
trade  space,  include  sensitivity  analysis,  and  consider  the  positive  and  negative  aspects  of 
each  option. 

•  The  results  of  the  AoAs  should  inform  requirements,  identify  cost-effective  solutions,  be 
transparent,  and  consider  operational  effectiveness. 

2.  What  do  you  think  the  major  issues  are  associated  with  AoAs  under  the  new  WSARA 
guidance? 
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•  OSD  CAPE  has  an  increased  level  of  responsibility  for  AoAs  under  WSARA  including: 
issuing  AoA  guidance;  detennining  if  the  AoA  was  perfonned  consistent  with  OSD 
CAPE’s  guidance;  and  recommending  AoA  updates. 

•  There  is  a  new  requirement  under  WSARA  to  perfonn  cost,  schedule,  and  performance 
trades.  AoAs  must  ensure  that  the  trade  space  is  explored  in  detail. 

•  WSARA  has  increased  the  visibility  of  AoAs,  which  has  resulted  in  a  greater  awareness 
and  interest  in  AoA  results.  The  JS,  the  Services,  Congress,  and  Government 
Accountability  Office  (GAO)  are  inquiring  about  the  results  of  AoAs. 

3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

•  An  AoA  is  required  at  MS  A  and  there  is  one  set  of  guidance,  which  is  good  for  all  MS 
decisions.  The  guidance  will  supply  questions  to  be  answered,  and  the  AoA  team  should 
answer  the  questions  to  their  best  ability  with  the  infonnation  available. 

•  An  AoA  does  not  have  to  be  updated  for  the  subsequent  MSs  unless  OSD  CAPE 
designates  an  update  is  required.  An  AoA  may  have  to  be  updated  for  the  following 
reasons:  (1)  the  underlying  assumption/facts  change,  (2)  a  new  threat  is  identified,  and  (3) 
the  program  changes  acquisition  categories  that  change  the  MDA  (e.g.,  from  the  Anny 
Acquisition  Executive  (AAE)  to  the  Defense  Acquisition  Executive  (DAE)). 

•  AoAs  may  be  required  at  later  MS  decisions  for  (1)  a  program  going  directly  from  MDD 
to  MS  B  or  post  MS  B,  or  (2)  a  program  that  changes  ACAT  levels. 

•  AoAs  should  be  tailored  to  the  specific  program  rather  than  for  the  MS.  The  analyses  are 
tailored  for  the  particular  program  rather  than  for  the  MS. 

4.  What  types  of  analysis  do  you  think  need  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 

•  The  analysis  should  be  quantitative  instead  of  qualitative  judgment-based.  Using  Subject 
Matter  Experts  (SMEs)  to  determine  how  much  they  desire  a  particular  attribute,  scoring 
the  results,  and  developing  weighing  factors  is  not  the  correct  process. 

•  The  analyses  must  be  understandable,  transparent,  and  support  the  conclusion. 

•  Exploratory  analysis  and  cost,  schedule,  and  perfonnance  trades  are  important 
components  to  an  AoA. 
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•  Sensitivity  analyses  should  be  performed  for  the  assumptions,  alternative  definitions,  and 
drivers.  For  the  “lesser”  alternatives,  sensitivity  analyses  should  be  perfonned  to 
detennine  what  it  would  take  to  make  it  a  more  competitive  alternative. 

•  The  analyses  should  identify  “tipping  points”  for  the  decision  choices.  If  there  are  no 
tipping  points  or  points  of  interest,  the  AoA  should  “stop  drilling,”  when  the  result  does 
not  make  a  distinction  between  alternatives. 

•  The  following  analyses  should  be  conducted  in  AoAs:  operational  effectiveness, 
operational  assessments,  performance,  cost,  sensitivity,  fully  burdened  cost  of  fuel,  and 
affordability/portfolio. 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

•  The  basic  alternatives  for  AoAs  include  a  baseline  of  what  is  currently  programmed  in 
service,  a  modification  of  what  is  currently  in  use,  a  NDI  system/foreign  system,  and  a 
new  start  system. 

•  There  should  be  a  broad  search  for  alternatives,  and  the  definition  of  those  alternatives 
should  be  kept  broad.  The  Service  should  not  presuppose  an  answer  and  prematurely 
reduce  the  decision  space. 

•  A  robust  screening  analysis  should  be  completed  upfront  and  presented  to  the  SAG, 
which  will  request  additional  excursions,  and  reframe  or  expand  the  analysis,  as 
necessary. 

•  The  alternatives  should  cover  the  decision  space,  and  that  trades  and  sensitivity  analyses 
should  be  performed.  A  good  screening  analysis  would  help  explore  the  trade  space. 

•  The  Services  should  follow  the  OSD  CAPE  study  guidance  when  developing  alternatives. 

6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

•  Foreign  systems  should  be  considered  in  AoAs,  and  the  Services  should  do  their  best  to 
obtain  data  and  infonnation.  Systems  from  threat  countries  should  not  be  considered. 

•  The  decision  to  actually  purchase  a  foreign  alternative  is  not  OSD  CAPE’s  or  the  AoA 
team’s  decision. 

•  The  cost  and  time  associated  with  modifying  foreign  alternatives  should  be  considered  in 
the  AoA. 

•  Another  purpose  of  considering  foreign  alternatives  is  to  learn  more  about  their  features, 
attributes,  technical  feasibility,  and  cost. 
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7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs ? 

•  The  tradeoff  analysis  should  explore  the  trade  space  and  decision  space  for  AoAs.  The 
boundaries  for  the  trade  space  should  be  expanded  to  ensure  a  robust  decision  space.  The 
trades  should  be  explored  throughout  the  AoA. 

•  AoAs  should  include  perfonnance  vs.  cost  curves. 

•  The  priority  of  tradeoff  factors  are  performance,  schedule,  and  then  cost. 

•  Operational  effectiveness  is  needed  in  the  tradeoff  analysis,  and  the  CAIG  should  be 
included  in  the  cost  estimates. 

•  AoA  should  inform  the  CDD;  although  the  CDD  should  not  be  written  to  make  one 
alternative  appear  better. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

•  OSD  CAPE  desires  more  transparency,  visibility,  understanding  of  the  analyses  and 
assumptions. 

•  AoAs  should  not  focus  on  one  preferred  alternative.  The  preferred  alternative  may  not  be 
attainable  or  viable,  and  a  program  based  on  this  alternative  may  be  canceled  in  later 
years. 

•  The  following  items  should  be  included  in  AoAs: 
o  Cost-effectiveness  analyses 

o  Real-world  test  results 

o  Real-world  foreign  alternatives 

o  Early  identification  of  cost  and  perfonnance  drivers 

o  Operational  effectiveness  to  cost  graphs 

o  Sensitivity  analyses  on  the  assumptions 

o  Consideration  of  the  resources  available,  requirements,  and  acquisition  process 

•  Cost  and  schedule  should  be  considered  when  developing  requirements. 

•  AoAs  should  be  streamlined  to  reduce  the  cost  of  conducting  the  AoAs. 
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Summary  of  Results 

This  chapter  provided  the  results  of  the  Army  and  OSD  CAPE  interviews.  A  summary  of 
interview  answers  for  each  of  the  eight  questions  were  provided  in  this  chapter.  The  Army  AoA 
team,  Anny  AoA  oversight  organizations,  and  OSD  CAPE  answers  were  provided  separately  so 
they  can  be  compared.  All  Army  and  OSD  CAPE  personnel  provided  detailed,  insightful 
answers  to  the  AoA  questions.  This  chapter  only  provided  a  summary  of  the  interview  results. 
Appendixes  A,  B,  and  C  provide  additional  information  to  the  summary  points  listed  in  the 
chapter. 
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CHAPTER  5 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  chapter  provides  a  discussion  of  the  interview  results,  comparison  of  interview 
results,  and  recommendations  for  improving  Army  AoAs.  The  discussion  is  divided  into  three 
sections,  including  discussions  of  the  Army  AoA  team  interview  results,  the  Army  AoA 
oversight  organizations  interview  results,  and  the  OSD  CAPE  interview  results.  After  these 
discussions,  the  interview  results  will  be  compared  across  the  three  groups.  Recommendations 
are  provided  to  better  meet  OSD  CAPE  expectations  and  to  better  support  the  materiel 
requirements  and  acquisition  processes.  The  chapter  also  includes  limitations,  implications  for 
further  research,  and  summary  and  conclusions. 

Discussion  of  Army  AoA  Team  Interview  Results 

Many  of  the  Army  AoA  team’s  purposes  of  AoAs  were  related  to  providing  information 
to  the  decision-makers.  The  Anny  AoA  team  noted  that  AoAs  were  to  frame  the  decision  space 
and  to  identify  the  best  investment  strategy  and  acquisition  option.  The  idea  of  comparing  best 
investment  courses  of  action  was  proposed.  Other  personnel  from  the  Anny  AoA  team  noted  that 
OSD  CAPE’s  decision  choices  are  not  necessarily  alternative-related. 

Anny  AoA  team  personnel  noted  OSD  CAPE’s  influence  has  increased  in  the  AoA 
process.  Some  noted  the  WSARA  focus  on  AoA  had  a  positive  impact  on  AoAs.  The  Anny  AoA 
team  said  AoAs  should  be  unbiased  with  no  preferred  solution  and  advocated  the  analyses  must 
be  transparent,  open,  and  understandable  to  OSD  CAPE.  Having  an  unbiased  AoA  with  no 
preferred  solution  also  was  a  theme  throughout  the  answers.  The  Anny  AoA  team  suggested  that 
OSD  CAPE  be  included  early,  and  often,  in  the  AoA  process.  By  working  more  closely  with 
OSD  CAPE,  OSD  CAPE’s  level  of  understanding  of  the  analysis  would  increase.  OSD  CAPE 
would  more  likely  believe  the  analysis  is  credible  and  relevant. 

The  Anny  AoA  team  personnel  provided  comments  on  the  timing  of  AoAs  and  suggested 
that  AoAs  be  conducted  at  each  MS  decision.  Benefits  would  include  an  increase  to  the  fidelity 
and  structure  of  the  analysis  and  an  increase  in  the  information  flow  for  the  analyses.  Anny  AoA 
team  personnel  noted  analyses  should  also  be  perfonned  before  MDD  to  identify  capability  gaps. 
These  analyses  may  not  be  associated  with  an  AoA. 

Anny  AoA  team  personnel  indicated  that  a  purpose  of  an  AoA  was  to  detennine  the 
desired  attributes  through  cost,  schedule,  and  perfonnance  trades.  The  Army  AoA  team  noted 


33 


that  an  alternative  should  represent  a  set  of  attributes,  and  the  alternative  should  not  be  associated 
with  a  system  or  platform.  Alternatives  are  vessels  to  hold  attributes  together  to  better  understand 
the  interdependencies  of  attributes. 

A  concern  was  raised  regarding  the  number  of  alternatives  considered  in  an  AoA.  The 
Anny  AoA  team  recommended  including  six  or  fewer  alternatives  in  an  AoA  to  be  able  to 
complete  the  analysis  in  a  9-  to  12-month  time  frame.  The  Army  AoA  team  noted  that 
alternatives  should  be  eliminated  if  they  are  infeasible,  and  there  should  be  a  limited  number  of 
“special  interest"  and  foreign  systems.  Special  interest  systems  denote  systems  that  are  required 
to  be  included  in  analyses  from  Army  or  OSD  senior  leaders.  The  Anny  AoA  team  also  noted 
that  alternatives  should  be  eliminated  by  the  SAG  if  the  alternative  are  no  longer  feasible  or  do 
not  add  information  to  the  decision  choices.  Also,  analyses  that  do  not  add  infonnation  relevant 
to  the  decision  should  be  stopped. 

Anny  AoA  team  personnel  were  very  concerned  about  obtaining  detailed  data  for  foreign 
systems.  The  Anny  AoA  team  noted  that  to  perform  the  same  type  analysis  on  foreign  systems 
as  U.S.  systems,  detailed  design,  perfonnance,  and  cost  data  would  be  required.  If  these  data 
cannot  be  obtained,  there  would  be  greater  uncertainty  in  the  results.  The  AoA  team  also  raised 
an  issue  that  all  foreign  systems  would  require  modifications,  and  the  cost  and  risk  of  these 
modifications  should  be  included  in  the  analyses.  Finally,  Army  AoA  personnel  said  that  the 
United  States  rarely  buys  foreign  systems,  and  that  foreign  systems  should  not  be  considered  if 
they  are  not  viable,  feasible  candidates. 

Anny  AoA  team  personnel  stated  that  AoAs  should  consider  and  focus  on  other  analyses 
besides  effectiveness.  These  other  areas  would  be  related  to  alternatives  impact  to  sustainment, 
force  structure,  transportability,  and  DOTMLPF. 

Tradeoff  analyses  for  cost,  schedule,  and  performance  were  noted  as  important  analyses 
in  AoAs.  Anny  AoA  team  personnel  noted  that  WSARA  required  the  analyses,  and  the  analyses 
should  be  performed  across  and  within  alternatives.  The  trades  should  be  conducted  at  the 
system  and  subsystem  levels.  The  AoA  team  said  the  trade  space  must  be  fully  explored,  and  the 
trades  should  continue  throughout  the  AoA.  Army  AoA  team  personnel  stated  that  an  AoA 
should  identify  the  most  affordable,  cost-effective  solution  through  the  tradeoff  and  cost-benefit 
analyses. 
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Risk  analysis  was  also  noted  as  an  important  analysis  in  an  AoA.  Army  AoA  team 
personnel  noted  that  risk  analyses  are  closely  related  to  the  tradeoff  analyses.  The  AoA  team 
recommended  that  risk  assessments  should  be  conducted  for  cost,  schedule,  and  technical  rather 
than  cost,  schedule,  and  perfonnance.  Army  AoA  team  personnel  stated  that  a  coherent,  robust 
risk  and  trade  methodology  must  be  developed  by  the  Army. 

Because  WSARA  increased  the  scope  and  complexity  of  AoAs,  the  Anny  AoA  team 
personnel  thought  there  was  a  significant  increase  in  analytical  work  to  address  WSARA  AoA 
requirements.  This  additional  workload  requires  additional  analysts  (cost  and  perfonnance)  and 
funding  to  support  conducting  AoAs.  There  are  resource  challenges  associated  with  the  conduct 
of  AoAs. 

Anny  AoA  team  personnel  suggested  that  O&S  and  LCC  be  used  in  AoAs — specifically 
for  the  tradeoff  analyses.  It  also  was  noted  that  cost  methodology  should  be  refined  to  develop 
more  estimating  relationships  that  link  perfonnance  to  cost,  incorporate  risk  better,  and  consider 
affordability. 

Discussion  of  Army  AoA  Oversight  Organizations  Interview  Results 

One  of  the  main  points  of  the  Army  AoA  oversight  personnel  was  that  AoAs  should 
support  the  requirements  development  process  and  indicated  that  WSARA  gave  OSD  CAPE 
more  influence  on  AoAs,  because  OSD  CAPE  has  to  declare  an  AoA  sufficient  (which  is  linked 
to  AoA  certification)  to  support  MS  decisions.  The  AoA  oversight  personnel  mentioned  that 
OSD  CAPE  is  more  focused  on  supporting  the  MS  decisions.  Anny  AoA  oversight  personnel 
stressed  that  AoAs  must  satisfy  all  the  customers  including  OSD  CAPE,  the  acquisition 
community,  the  requirements  community,  and  the  portfolio  mix  community. 

Anny  AoA  oversight  personnel  said  AoAs  must  address  the  decision  process  and  OSD 
CAPE’s  decision  choices,  and  noted  OSD  CAPE’s  decision  choices  are  more  strategically 
focused.  AoA  oversight  personnel  also  recommended  that  AoAs  not  address  issues  already  being 
addressed  in  the  acquisition  process  (e.g.,  source  selection,  acquisition  strategy,  designing  the 
system). 

Another  point  from  the  Anny  AoA  oversight  personnel  was  that  analyses  should  be 
conducted  before  the  MDD.  These  analyses  would  entail  the  identification  of  capability  gaps, 
threats,  problems,  attribute  trade  space,  and  feasible  alternatives.  The  AoA  oversight  personnel 
also  noted  that  capability  gaps  and  CONOPS  should  be  developed  before  the  start  of  an  AoA. 
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Army  AoA  oversight  personnel  stressed  that  the  purpose  of  an  AoA  is  to  develop  the 
supporting  evidence  to  identify  an  achievable,  affordable,  and  operationally  relevant  set  of 
attributes.  The  AoA  oversight  personnel  believe  that  alternatives  are  used  to  represent  a  set  of 
attributes,  which  would  be  used  in  developing  the  CDD. 

Some  of  the  Anny  AoA  oversight  personnel  also  noted  that  timing  of  AoAs  is  not  well 
coordinated  with  the  requirements  development  process  or  to  support  the  PMs.  They  stated  that 
an  AoA  for  MS  A  is  too  early,  because  not  enough  information  is  available  to  make  the  cost, 
schedule,  and  perfonnance  trades.  MS  B  is  too  late,  because  the  CDD  must  be  finalized  before 
then.  These  AoA  oversight  personnel  recommended  the  correct  timing  for  an  AoA  is  at  the  PDR. 
In  addition  to  the  MS  A  or  PDR  AoA,  the  Army  AoA  oversight  personnel  noted  that  AoAs,  or 
AoA  updates,  are  needed  to  support  all  decision  points  throughout  the  acquisition  process. 

Because  Army  AoA  oversight  personnel  believed  that  foreign  systems  may  not  be 
purchased  by  the  United  States,  the  oversight  personnel  thought  that  foreign  systems  should  only 
be  considered  in  AoAs  for  comparison  purposes  to  investigate  capabilities  and  attributes.  The 
AoA  oversight  personnel  also  noted  that  foreign  systems  would  have  to  be  modified  to  meet  U.S. 
standards. 

The  Anny  AoA  oversight  personnel  stressed  the  importance  of  tradeoff  analyses  for  cost, 
schedule,  and  perfonnance  and  said  the  trades  should  be  performed  within  and  among 
alternatives  to  address  the  decision  space.  The  oversight  personnel  noted  that  risk  assessments 
should  also  be  made  on  the  given  cost,  schedule,  and  performance,  or  the  traded  values.  Also 
related  to  the  tradeoff  analyses,  the  Anny  AoA  personnel  noted  that  the  importance  of  reducing 
system  costs  for  affordability  reasons. 

Because  of  the  increased  workloads  of  the  analytical  community  from  post-WSARA 
AoAs,  the  Army  AoA  oversight  personnel  thought  there  would  be  a  shortfall  of  resources 
(funding  or  personnel).  They  thought  a  central  funding  source  should  be  available  for  AoAs,  and 
they  noted  the  need  for  more  skilled  analysts  for  technical  risk  assessment,  cost  analyses,  and 
affordability  assessments. 

The  Anny  AoA  oversight  personnel  recommended  that  the  Anny  be  more  candid  and 
open  with  OSD  CAPE  and  thought  the  Army  should  work  closer  with  OSD  CAPE  earlier  in  the 
AoA  processes  to  understand  guidance  and,  obtain  agreement  on  the  alternatives  and  foreign 
systems. 
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Discussion  of  OSD  CAPE  Interview  Results 

OSD  CAPE  personnel  are  focused  on  supporting  acquisition  MS  decisions.  WSARA 
requires  an  AoA  to  be  performed  before  a  MS  decision  in  accordance  with  OSD  CAPE  guidance. 
OSD  CAPE  personnel  noted  that  their  level  of  responsibility  for  an  AoA  increased  due  to 
WSARA.  OSD  CAPE  develops  the  AoA  guidance  and  determines  if  the  AoA  was  performed 
consistently  with  the  guidance.  OSD  CAPE  noted  that  WSARA  also  has  increased  the 
importance  and  interest  in  AoAs. 

A  common  theme  throughout  OSD  CAPE’s  responses  was  their  desire  for  the  AoAs  to  be 
objective,  transparent,  and  open.  OSD  CAPE  personnel  don’t  want  the  Army  to  have  a  preferred 
alternative  at  the  beginning  of  an  AoA  and  for  the  Anny  to  gear  the  analyses  toward  that 
solution.  They  would  like  an  objective  analysis  that  supports  a  recommended  solution.  OSD 
CAPE  personnel  would  like  the  AoAs  to  consider  the  positive  and  negative  aspects  of  each 
option.  OSD  CAPE  personnel  also  want  a  better  understanding  of  the  analyses  and  assumptions 
and  would  like  the  Anny  to  be  more  transparent  with  their  assumptions  and  analyses. 

It  is  OSD  CAPE’s  position  that  an  AoA  is  required  at  MS  A,  and  there  is  one  set  of 
guidance  good  for  all  MS  decisions.  The  AoA  guidance  supplies  questions  to  be  answered,  and 
OSD  CAPE  noted  that  the  AoA  team  should  answer  the  questions  to  its  best  ability  with  the 
information  available.  This  is  especially  germane  to  a  MS  A  AoA,  where  some  infonnation  may 
not  be  available.  OSD  CAPE  personnel  also  stressed  that  AoAs  are  not  required  at  later  MS 
decisions,  except  when  they  request  an  update.  An  update  could  be  requested  if  assumptions 
change,  new  threats  are  identified,  or  the  ACAT  level  changes. 

OSD  CAPE  is  interested  in  having  a  robust  set  of  alternatives  considered  in  the  AoA, 
such  as  a  broad  search  for  alternatives  including  a  detailed  screening  analysis.  OSD  CAPE 
personnel  would  like  the  alternatives  to  cover  the  decision  space,  and  they  are  also  interested  in 
changing  “lesser”  alternatives  to  detennine  if  they  could  be  made  more  competitive. 

OSD  CAPE  personnel  strongly  supported  having  foreign  systems  in  the  AoA.  They 
realized  the  foreign  system  might  not  be  purchased,  but  they  did  not  think  that  was  a  reason  not 
to  consider  them.  OSD  CAPE  personnel  noted  that  the  decision  actually  to  purchase  a  foreign 
alternative  does  not  fall  to  OSD  CAPE  nor  the  AoA  team.  They  would  like  the  Army  to  do  its 
best  to  obtain  data  and  infonnation  on  the  foreign  systems.  OSD  CAPE  suggested  another 
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purpose  of  considering  foreign  alternatives  is  to  learn  more  about  their  features,  attributes, 
technical  feasibility,  and  cost. 

OSD  CAPE  personnel  stressed  that  analysis  should  be  quantitative  instead  of  qualitative 
judgment-based.  They  do  not  like  the  SME-based  value  assessments,  but  rather  have  analyses 
based  on  engineering  models  or  real-world  test  results.  OSD  CAPE  personnel  noted  that  the 
analyses  should  be  focused  on  the  decision  choices,  and  there  is  no  point  in  performing  analyses 
that  do  not  support  the  decision  choices. 

OSD  CAPE  personnel  would  like  the  AoA  to  explore  the  trade  space  in  detail.  They 
require  cost,  schedule,  and  performance  trades  to  be  performed  in  the  AoAs  and  noted  that 
sensitivity  analyses  should  be  conducted  along  with  the  tradeoff  analyses,  and  the  bounds  for  the 
sensitivity  analyses  should  be  expanded. 

Another  theme  throughout  the  OSD  CAPE  interviews  was  OSD  CAPE  wanted  to  see 
affordable  and  cost-effective  alternatives;  OSD  CAPE  desired  cost-effectiveness  analyses, 
operational  effectiveness  to  cost  graphs,  and  performance  vs.  cost  graphs. 

Comparison  of  Interview  Results 

A  number  of  similarities  and  a  few  differences  existed  between  the  three  groups  of 
interview  results.  All  groups  agreed  that  WSARA  has  increased  OSD  CAPE  influence  in  AoAs, 
and  that  OSD  CAPE  is  focused  on  supporting  the  MS  decisions.  The  Army  personnel,  especially 
the  Anny  AoA  oversight  group,  wanted  the  AoA  process  to  increase  their  support  to  the 
requirements  development  and  acquisition  communities. 

Both  Army  and  OSD  CAPE  personnel  desired  more  communication  between  OSD  CAPE 
and  the  Anny  analysis  community,  and  this  communication  should  start  earlier  and  continue 
throughout  the  AoA.  All  groups  agreed  that  the  Anny  should  be  more  candid,  open,  and 
transparent  with  OSD  CAPE.  Another  important  point  was  the  objectivity  of  the  analysis  and 
presenting  results.  Army  and  OSD  CAPE  that  there  should  not  be  a  preferred  alternative,  and  the 
analyses  should  be  unbiased.  Finally,  there  was  agreement  that  the  analyses  and  assumptions 
must  be  understandable  to  OSD  CAPE,  which  could  be  realized  through  a  closer  working 
relationship. 

Anny  personnel  stressed  the  importance  of  providing  infonnation  to  the  decision-makers 
and  addressing  OSD  CAPE’s  decision  choices.  The  Anny  noted  that  OSD  CAPE’s  decision 
choices  may  not  be  related  to  only  the  alternatives,  but  to  investment,  acquisition,  and 
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strategic/fleet  options.  OSD  CAPE  personnel  stated  that  they  also  wanted  other  questions 
answered  in  addition  to  the  comparison  of  alternatives.  OSD  CAPE  personnel  noted  that  the 
analyses  should  be  focused  on  the  decision  choices,  and  analyses  should  not  be  performed  unless 
they  support  the  decision  choices.  Anny  AoA  team  personnel  said  decision  choices  may  focus  on 
other  issues  besides  effectiveness,  such  as  sustainment,  force  structure,  transportability,  and 
DOTMLPF. 

There  were  different  thoughts  on  the  number  and  timing  of  AoAs  by  Anny  and  OSD 
CAPE  personnel.  OSD  CAPE’s  position  is  that  one  AoA  is  required  (usually  at  MS  A),  and 
AoAs  are  not  required  at  later  MS  decisions  unless  an  update  is  requested.  The  Anny  AoA  team 
personnel  stressed  that  AoAs  are  needed  at  each  MS  decision.  The  AoA  team  thought  that 
without  a  fonnal  AoA,  the  fidelity  and  structure  of  the  analyses  would  suffer,  and  it  would  be 
more  difficult  to  obtain  the  needed  infonnation  to  perfonn  analyses  without  fonnal  AoAs.  Both 
Anny  AoA  oversight  and  team  personnel  recommended  that  analyses  should  be  performed 
before  MDD  to  identify  capability  gaps,  potential  issues,  and  potential  feasible  solutions.  These 
pre-MDD  analyses  are  necessarily  part  of  an  AoA.  Some  Army  AoA  oversight  personnel  also 
wanted  to  change  the  timing  of  AoAs  to  better  support  the  requirements  development  process 
and  the  PMs.  They  suggested  a  better  timing  for  an  AoA  is  at  PDR. 

Anny  personnel  stressed  the  importance  of  identifying  a  required  set  of  attributes  for  a 
system  in  the  AoA  process,  especially  for  the  MS  A  AoA.  They  said  the  AoA  should  develop  the 
supporting  evidence  to  identify  an  achievable,  affordable,  and  operationally  relevant  set  of 
attributes.  To  follow  that  objective,  they  noted  that  an  alternative  should  represent  a  set  of 
attributes  rather  than  be  associated  with  a  system  or  platform.  The  Army  personnel  said  these 
attributes  would  be  used  to  help  develop  the  CDD.  OSD  CAPE  mentioned  that  the  AoA  should 
help  infonn  requirements,  but  did  not  discuss  attributes  in  detail. 

The  Army  AoA  team  raised  the  concern  about  having  too  many  alternatives  in  an  AoA 
and  recommended  having  six  or  fewer  alternatives.  They  also  wanted  alternatives  eliminated  by 
the  SAG  if  they  were  no  longer  feasible  or  add  nothing  to  the  decision  choices.  OSD  CAPE,  on 
the  other  hand,  wanted  a  robust  set  of  alternatives  in  the  AoA.  They  wanted  a  broad  search  in  the 
beginning  and  the  range  of  alternatives  to  cover  the  decision  space. 

The  views  of  foreign  alternatives  differed  between  the  Anny  and  OSD  CAPE.  OSD 
CAPE  personnel  strongly  supported  included  foreign  alternatives  in  the  AoAs.  Army  AoA  team 
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personnel  were  concerned  about  obtaining  data  for  foreign  systems  and  performing  equivalent 
analysis  between  foreign  and  U.S.  alternatives.  OSD  CAPE  personnel  said  the  Army  AoA  team 
should  do  its  best  to  obtain  data  on  the  foreign  systems.  Anny  and  OSD  CAPE  personnel  agreed 
that  the  foreign  alternative  may  not  be  selected,  because  of  political  reasons.  OSD  CAPE 
personnel  thought  purchasing  a  foreign  system  was  not  their  decision,  and  that  foreign  systems 
still  should  be  considered.  Anny  personnel  recommended  not  including  foreign  systems  that  are 
not  viable,  feasible  candidates.  Army  AoA  oversight  personnel  thought  foreign  systems  should 
only  be  considered  in  AoAs  for  comparison  purposes  to  investigate  capabilities  and  attributes. 
Both  Army  and  OSD  CAPE  personnel  noted  that  modifications  would  have  to  be  made  to 
foreign  systems,  and  the  cost  and  schedule  of  these  modifications  should  be  included  in  the  AoA. 

There  was  agreement  between  the  Army  and  OSD  CAPE  on  tradeoff  analyses.  Both  the 
Anny  and  OSD  CAPE  stressed  the  importance  of  tradeoff  analyses  for  cost,  schedule,  and 
perfonnance.  All  groups  said  trades  should  be  performed  within  and  among  alternatives  to 
address  the  decision  space  and  explore  the  trade  space.  Army  and  OSD  CAPE  personnel  agreed 
that  the  AoA  should  identify  affordable,  cost-effective  solutions.  Army  personnel  noted  the 
importance  of  risk  analyses  in  the  AoA.  OSD  CAPE  did  not  stress  the  importance  of  risk 
analyses.  However,  they  did  recommend  risk  analyses  in  their  publications. 

Anny  personnel  from  the  AoA  team  and  AoA  oversight  organizations  discussed  the 
increase  in  analytical  work  because  of  WSARA.  The  Army  noted  challenges  associated  with 
resources,  and  recommended  that  a  central  funding  site  be  available  to  conduct  AoAs.  The  Anny 
personnel  also  identified  the  need  for  more  skilled  analysts  in  the  technical  risk  assessment,  cost 
analyses,  and  affordability  areas. 

Recommendations 

The  following  are  key  recommendations  based  on  the  interview  answers  and  literature 
search  results.  These  recommendations  are  to  better  meet  OSD  CAPE  expectations,  meet  other 
AoA  customer  expectations,  and  improve  the  value  of  AoAs.  Appendix  D  provides  additional 
recommendations. 

The  Army  AoA  community  and  OSD  CAPE  should  build  a  closer  working  relationship.  In 
the  past,  the  relationship  between  the  Army  and  OSD  CAPE  was  characterized  by  suspicion  and 
distrust.  The  current  relationship  has  improved,  but  both  organizations  should  continue  to  build  a 
closer  working  relationship  built  on  trust.  As  recommended  by  both  organizations,  there  should 
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be  more  communication  between  the  Anny  AoA  community  and  OSD  CAPE.  Both  the  Army 
and  OSD  CAPE  must  be  candid,  open,  and  transparent  in  their  interactions.  Trust  has  to  be 
extended  to  build  trust,  and  both  organizations  must  demonstrate  respect  for  each  other  (Covey, 
2006).  The  Army  needs  to  ensure  that  the  analyses  and  results  are  objective  and  appear  objective 
to  OSD  CAPE.  Both  organizations  need  to  interact  frequently  to  ensure  OSD  CAPE  has  an 
understanding  of  the  assumptions  and  analyses.  The  Anny  AoA  team  and  OSD  CAPE  personnel 
should  discuss  the  analyses  monthly  either  through  meetings,  video  teleconferences,  or 
teleconferences.  Regular  reviews  at  the  SES  level  should  be  conducted  to  ensure  the  Anny  team 
is  on  track  to  meet  OSD  CAPE  expectations.  The  benefits  of  a  closer  working  relationship  would 
be  less  rework  because  of  clarified  expectations,  greater  acceptance  of  the  analyses  by  OSD 
CAPE,  fewer  OSD  CAPE  questions  later  regarding  the  analyses,  less  follow-up  work,  and  more 
focused  analytical  products. 

AoAs  should  focus  on  the  decision  choices  and  the  decision  space.  In  the  past,  Anny 
AoAs  have  focused  on  performing  specific  analyses  on  well-defined  alternatives.  The  analyses 
tended  to  center  on  operational  effectiveness  and  performance,  and  there  were  only  a  few,  well- 
defined,  obvious  alternatives.  WSARA,  and  the  reduction  of  DoD  funding,  caused  a  change  in 
the  information  decision-makers  require.  There  is  more  focus  on  cost,  affordability,  and  portfolio 
management.  There  is  more  scrutiny  on  starting  a  new  program  or  on  how  many  items  are 
procured.  The  decision  choices  may  be  related  not  only  to  the  alternatives,  but  to  investment, 
acquisition,  and  strategic/fleet  options.  The  decision  choices  may  focus  on  other  issues  besides 
effectiveness,  such  as  DOTMLPF,  sustainment,  force  structure,  transportability,  and  strategic 
impacts.  Another  important  factor  is  the  decision  space  (i.e.,  the  breadth  of  options  to  consider). 
The  trade  space  for  cost,  schedule,  and  perfonnance  must  be  fully  explored  to  frame  the  decision 
space.  Due  to  funding  restraints,  the  decision  space  may  have  to  be  widened  to  weigh  options  not 
typically  considered.  The  AoA  team  should  work  closely  with  OSD  CAPE  and  Army  leaders  to 
fully  understand  the  decision  choices  and  the  decision  space.  The  types  of  analyses  should  be 
focused  on  the  decision  choices.  Analysis  that  does  not  support  the  decision  choices  should  not 
be  performed.  This  focus  could  result  in  less  operational  effectiveness  and  perfonnance  analyses 
in  AoAs,  depending  on  the  specific  decision  choices.  A  focus  on  the  decision  choices  and 
decision  space  would  increase  the  relevant,  usable  information  presented  to  the  decision-makers. 
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AoA.s  should  identify  an  achievable,  affordable,  and  operationally  relevant  set  of 
attributes.  Besides  supporting  the  MS  decisions,  AoAs  should  also  infonn  the  requirement 
development  process.  MS  A  AoAs  should  identify  specific  attributes  needed  to  fill  the  capability 
gaps,  which  would  feed  into  the  CDD  development.  AoAs  should  develop  the  supporting 
evidence  to  identify  an  achievable,  affordable,  and  operationally  relevant  set  of  attributes.  The 
alternatives  should  represent  a  set  of  attributes  and  should  not  be  related  to  a  certain  system  or 
platform  (i.e.,  the  alternatives  should  be  platfonn  agnostic).  If  OSD  CAPE’s  decision  choices  are 
related  to  certain  systems  or  platforms,  the  AoA  would  have  to  include  them.  Again,  the  AoA 
team  must  work  closely  with  Army  and  OSD  CAPE  leaders  to  ensure  the  alternatives  address  the 
decision  choices.  The  CDD  development  process  would  greatly  benefit  from  the  identification  of 
an  achievable,  affordable,  and  operationally  relevant  set  of  attributes. 

The  focus  of  foreign  systems  should  be  to  investigate  features,  capabilities,  attributes, 
feasibility,  and  cost.  Foreign  systems  will  be  required  in  AoAs  through  OSD  CAPE  guidance. 
Although  the  foreign  system  may  not  be  pursued  in  the  acquisition  process,  there  is  much  to  be 
learned  from  analyzing  foreign  systems.  Foreign  systems  may  include  unique  features  or 
attributes  not  included  in  U.S.  systems.  By  examining  these  features  or  attributes,  the  Army  may 
include  them  in  its  CDD  development.  If  a  particular  technology  is  used  on  a  foreign  system, 
information  on  the  feasibility  of  including  that  technology  on  U.S.  systems  would  be  gained. 
Identifying  the  cost  of  the  foreign  systems  would  provide  a  cost  comparison  for  U.S.  systems.  It 
may  be  easier  to  identify  the  attributes  and  capabilities  of  foreign  vehicle  rather  than  obtain  the 
detailed  design  data.  This  focus  on  capabilities  and  attributes  could  reduce  the  data  collection 
issues. 

The  Army  should  create  a  formal  analytical  process  that  supports  MS  decisions, 
requirements  development,  materiel  acquisition,  and  portfolio  management  throughout  the 
acquisition  process.  The  analytical  process  must  contain  an  AoA  for  MS  A,  or  the  first  MS 
decision  for  a  program,  per  statutory  requirements.  OSD  CAPE  provides  fonnal  guidance  for  this 
AoA.  The  Army  can  perform  additional  AoAs,  or  AoA  updates,  at  the  MS  B  and  MS  C 
decisions,  but  they  may  not  have  OSD  CAPE  oversight.  This  analytical  process  should  start  pre- 
MDD  by  supporting  capability  gap  development,  attribute  trade  space,  alternative  fonnulation, 
and  issue  identification.  An  AoA,  a  Business  Case  Analysis  (BCA),  or  a  Cost-Benefit  Analysis 
(C-BA)  can  be  performed  to  support  the  CDD  development.  These  analyses  can  leverage  the  MS 
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A  AoA  and  fonn  the  basis  for  a  MS  B  AoA.  The  timing  for  this  analysis  would  be  for  the  PDR, 
and  the  tradeoff  and  risk  analyses  would  be  beneficial  to  the  PMs.  Army  tradeoff  and  risk 
analysis  would  help  the  PM  assess  the  contractors  design  at  PDR.  For  MS  B,  the  AoA  could 
compare  the  alternatives  to  the  requirements  and  to  each  other  and  help  detennine  whether  to 
build  the  system.  Post-MS  B,  analyses  could  support  production,  fielding,  distribution,  and 
sustainment  decisions.  An  AoA  team  could  be  established  pre-MDD  and  continue  throughout  the 
acquisition  process.  The  benefits  of  a  formal  analytical  process  include  the  following:  increased 
structure  of  analyses;  greater  fidelity  of  analyses;  increased  support  to  the  requirements 
development,  PEO/PM,  and  portfolio  management  communities;  and  retention  of  experienced 
analyst  for  a  specific  system. 

Limitations 

Interviews  were  conducted  for  the  key  personnel  at  AMSAA,  TRAC,  DASA-CE, 
DAMO-CI,  and  ARCIC.  If  time  were  not  constrained,  it  would  have  been  beneficial  to  interview 
Anny  personnel  from  the  ARL  Survivability  and  Lethality  Analysis  Directorate  (SLAD),  the 
Office  of  the  ASA(ALT),  and  the  Office  of  the  DCS  G-8.  Other  organizations  where  it  could  be 
potentially  beneficial  to  interview  personnel  are  the  TRADOC  COEs,  RDECs,  PEOs,  and  PMs. 
OSD  CAPE  was  the  only  organization  where  personnel  were  interviewed  for  OSD.  Other 
potentially  beneficial  organization’s  personnel  to  interview  are  with  the  USD(AT&L)  and  the  JS. 
Implications  for  Future  Research 

Future  research  could  expand  the  scope  of  organizations  to  be  interviewed.  As  noted  in 
the  previous  section,  personnel  from  other  Army  and  OSD  organizations  could  be  interviewed. 
Besides  interviewing  additional  people,  future  research  could  focus  on  studying  the  Army,  Air 
Force,  and  Navy  AoAs  that  have  been  deemed  successful  by  OSD  CAPE.  A  broader  examination 
of  how  foreign  countries  conduct  AoA-like  analyses  may  be  beneficial.  Also,  a  historical 
inspection  of  analyses  was  conducted  before  AoA  could  be  the  focus  of  future  research.  Future 
research  also  could  be  perfonned  on  the  creation  of  a  fonnal  analytical  process  that  supports  MS 
decisions,  requirements  development,  materiel  acquisition,  and  portfolio  management. 

Summary  and  Conclusions 

This  report  outlined  the  research  to  detennine  how  to  improve  Army  AoAs  to  better  meet 
the  requirements  of  the  WSARA  of  2009  and  the  expectations  of  OSD  CAPE.  A  literature 
review  was  performed  to  provide  an  overview  of  the  statutory  and  regulatory  requirements  for 


43 


AoAs,  the  aspects  of  WSARA  related  to  AoA,  Army  efforts  to  improve  AoAs,  and  OSD  CAPE 
suggestions  on  how  to  improve  AoAs.  Results  of  the  literature  search  were  provided  in  Chapter 
2.  Chapter  3  reviewed  the  research  methodology,  identified  the  interviewees,  and  listed  the 
interview  questions.  Key  senior-level  Army  and  OSD  CAPE  personnel  were  interviewed,  and 
their  key  points  presented  in  Appendixes  A,  B,  and  C.  From  the  interviews,  a  summary  of  the 
key  points  was  compiled  and  presented  in  Chapter  4.  Chapter  5  included  discussions  of  Anny 
AoA  team,  Army  AoA  oversight  organizations,  and  OSD  CAPE  interview  results.  These  results 
of  the  three  different  groups  were  compared.  Recommendations  were  developed  from  the 
interview  results  to  improve  Anny  AoAs.  These  recommendations  included  the  following: 

•  The  Anny  AoA  community  and  OSD  CAPE  should  build  a  closer  working  relationship. 

•  AoAs  should  focus  on  the  decision  choices  and  the  decision  space. 

•  AoAs  should  identify  an  achievable,  affordable,  and  operationally  relevant  set  of 
attributes. 

•  The  focus  of  foreign  systems  should  be  to  investigate  features,  capabilities,  attributes, 
feasibility,  and  cost. 

•  The  Anny  should  create  a  formal  analytical  process  that  supports  MS  decisions, 
requirements  development,  materiel  acquisition,  and  portfolio  management  throughout 
the  acquisition  process. 
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GLOSSARY  OF  ACRONYMS  AND  TERMS 

Army  Acquisition  Executive 
Acquisition  Category 
Acquisition  Decision  Memorandum 
Anny  Materiel  Systems  Analysis  Activity 
Average  Manufacturing  Unit  Cost 
Analysis  of  Alternatives 
Average  Procurement  Unit  Cost 
Anny  Research  Laboratory 

Assistant  Secretary  of  the  Anny  for  Acquisition,  Logistics,  and  Technology 
Acquisition,  Technology  and  Logistics 
Business  Case  Analysis 
Basis  of  Issue 

Command,  Control,  Communications,  and  Computers 

Center  for  Anny  Analysis 

Computer-Aided  Design 

Cost  Analysis  Improvement  Group 

Cost  Assessment  and  Program  Evaluation 

Cost  Analysis  Requirements  Description 

Cost -Benefit  Analysis 

Capabilities  Assessment 

Chemical,  Biological,  Radiological,  and  Nuclear 
Capability  Development  Document 
Capabilities,  Development  and  Integration  Directorate 
Center  of  Excellence 

Cost  and  Operational  Effectiveness  Analysis 
Colonel 

Concept  of  Operations 
Commercial-Off-The-Shelf 
Critical  Technology  Elements 
Defense  Acquisition  Board 
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DASA-CE 

DCAPE 

DCS 

DDR&E 

DFAR 

DFARS 

DoD 

DoDD 

DoDI 

DOTMLPF 

DPA&E 

DTM 

DUSA-OR 

FLVN 

GAO 

GFE 

GS 

HQDA 

ICD 

ICDT 

ICE 

IPR 

JCIDS 

JIAT 

JROC 

JS 


Defense  Acquisition  Executive 
Defense  Acquisition  Guidebook 

Department  of  the  Army,  Military  Operations  -  Capabilities  Integration 
Deputy  Assistant  Secretary  of  the  Army  for  Cost  and  Economics 
Director,  Cost  Assessment  and  Program  Evaluation 
Deputy  Chief  of  Staff 

Director,  Defense  Research  and  Engineering 

Defense  Federal  Acquisition  Regulation 

Defense  Federal  Acquisition  Regulation  Supplement 

Department  of  Defense 

Department  of  Defense  Directive 

Department  of  Defense  Instruction 

Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel 
and  Facilities 

Director,  Program  Analysis  &  Evaluation 
Directive-Type  Memorandum 

Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

Fort  Leavenworth 

Government  Accountability  Office  (formerly  known  as  the  General  Accounting 
Office) 

Government-Furnished  Equipment 
General  Schedule 

Headquarters  Department  of  the  Anny 
Initial  Capabilities  Document 
Integrated  Capabilities  Development  Team 
Independent  Cost  Estimate 
In-Progress  Review 

Joint  Capabilities  Integration  and  Development  System 

Joint  Integrated  Analysis  Tool 

Joint  Requirements  Oversight  Council 

Joint  Staff 
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JUONS 

Joint  Urgent  Operational  Needs  Statement 

KPP 

Key  Performance  Parameter 

KSA 

Key  System  Attribute 

LCC 

Life  Cycle  Cost 

MAIS 

Major  Automated  Information  Systems 

MDA 

Materiel  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Programs 

MDD 

Materiel  Development  Decision 

MS 

Milestone 

MSA 

Materiel  Solutions  Analysis 

O&S 

Operations  and  Support 

OSD 

Office  of  Secretary  of  Defense 

OTS 

Off  the  Shelf 

PDR 

Preliminary  Design  Review 

PEO 

Program  Executive  Officer 

PIP 

Product  Improvement  Program 

PM 

Program  Manager 

POM 

Program  Objective  Memoranda 

RDEC 

Research,  Development,  and  Engineering  Center 

RDECOM 

Research,  Development,  and  Engineering  Command 

RFP 

Request  for  Proposal 

SAG 

Senior  Advisory  Group 

SES 

Senior  Executive  Service 

SLAD 

Survivability  and  Lethality  Analysis  Directorate 

SME 

Subject  Matter  Expert 

SSA 

Source  Selection  Authority 

ST&E 

Science,  Technology  and  Engineering 

SWAP 

Size,  Weight,  And  Power 

TAA 

TD 

Technology  Development 

TRAC 

TRADOC  Analysis  Center 
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TRADOC  Training  and  Doctrine  Command 

TRL  Technology  Readiness  Level 

U.S.  United  States 

U.S.C.  United  States  Code 


USD(AT&L)  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
WSARA  Weapon  Systems  Acquisition  Refonn  Act  of  2009 
WSMR  White  Sands  Missile  Range 
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APPENDIX  A 

ARMY  AOA  TEAM  INTERVIEW  RESULTS 

This  section  provides  the  answers  for  the  interviewees  from  AMSAA,  TRAC,  and 
DASA-CE,  including  the  following: 

W.  Crain,  personal  communication,  January  17,  2012; 

M.  Bauman,  personal  communication,  April  3,  2012; 

P.  O’Neill,  personal  communication,  January  20,  2012; 

P.  Blechinger  &  B.  Pippin,  personal  communication,  March  9,  2012; 

G.  Lambert,  personal  communication,  March  14,  2012; 

S.  Barth,  D.  Henningsen,  &  M.  Anvari,  personal  communication,  March  19,  2012; 

D.  Hersh,  personal  communication,  March  16,  2012; 

R.  Chandler,  personal  communication,  January  20,  2012; 

F.  Decker,  personal  communication,  February  8,  2012; 

R.  Eaton,  personal  communication,  March  16,  2012; 

S.  Schoeb,  personal  communication,  March  5,  2012; 

A.  Dinsmore,  personal  communication,  December  27,  2011;  and 

S.  Vessey  &  A.  Tichenor,  personal  communication,  March  30,  2012. 

These  organizations  perform  analyses  for  AoAs  and  fonn  the  core  AoA  team.  The 
answers  are  compiled  for  each  question.  For  each  question,  the  interviewees’  answers  are 
grouped  by  general  ideas.  A  summary  of  the  key  points  for  all  interviewees  are  provided 
followed  by  the  interviewees’  specific  comments  as  bullet  points.  The  number  of  bullet  points 
may  not  align  to  the  number  of  people  interviewed,  because  some  interviewees  may  have 
provided  multiple  comments,  or  no  comments,  related  to  the  summarized  point.  Names  are  not 
associated  with  specific  comments  to  ensure  nonattribution. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

The  purpose  of  AoAs  is  to  frame  the  decision  space  and  to  identify  the  best  investment 
strategy  and  acquisition  option.  There  is  no  best  alternative,  but  rather  a  best  acquisition 
decision. 

•  The  purpose  of  an  AoA  is  to  frame  the  decision  space  associated  with  a  major 
acquisition  decision. 
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•  In  the  past,  before  WSARA,  AoAs  compared  alternatives  to  detennine  the  best 
alternative.  For  current  AoAs,  the  purpose  is  to  frame  the  real  decision — there  is  no 
best  alternative,  but  rather  a  best  acquisition  decision.  AoAs  should  look  at  new 
equipment  in  tenns  of  the  best  investment  strategy;  assessment  such  as  marginal 
return  on  investments  should  be  included. 

•  AoAs  provide  comparison  of  possible  options.  Options  can  be  defined  as  acquisition 
options,  and  the  alternatives  are  a  new  start  system,  modified  system,  etc.  The 
purpose  of  AoAs  are  not  to  just  compare  alternatives;  AoAs  should  also  define  a  set 
of  pluses  and  minuses  for  each  alternative,  inform  decision-makers  on  the  acquisition 
options,  and  identify  how  to  fulfill  needs/gaps  with  a  materiel  solution.  AoAs  are 
more  than  just  a  “buy  this  system”  assertion. 

The  purpose  of  AoAs  is  to  inform  the  decision-makers  on  best  investment  courses  of 

action. 

•  The  purpose  of  AoAs  is  to  inform  the  decision-makers  on  different  investment 
courses  of  action  for  meeting  or  resolving  capability  gaps.  The  investment  strategies 
are  tied  to  an  amount  of  funding  across  the  life  cycle.  AoAs  should  detennine  the  best 
alternative  at  an  affordable  price  that  mitigates  the  force  capability  gap. 

The  purpose  of  an  AoA  is  to  identify  the  most  affordable,  cost-effective  solution  through 
tradeoff  and  C-BA. 

•  An  AoA  is  a  C-BA  of  acquisition  alternatives  to  help  decision-makers  select  the  most 
appropriate  alternative  for  production.  The  AoA  is  a  modem-day  Cost  and 
Operational  Effectiveness  Analysis  (COE A). 

•  The  purpose  of  an  AoA  is  to  identify  the  most  cost-effective  solution  that  is 
affordable. 

•  There  should  be  trades  during  the  MS  A  AoA  related  to  cost-effectiveness  and 
affordability. 

•  The  purpose  of  an  AoA  is  to  select  the  best  alternative  to  meet  requirements. 

•  The  AoA  is  used  to  determine  the  most  cost-effective  solution  to  meet  a  given 
requirement  or  set  of  requirements. 

The  purpose  of  AoAs  is  to  inform  decision-makers  on  benefits  of  materiel  solutions  to 
meet  capability  gaps. 
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•  The  purpose  of  AoAs  is  to  in forrn  decision-makers  on  the  benefits  of  bringing  another 
system  into  service.  AoAs  assess  the  benefits  of  materiel  solutions,  compare 
performance/cost  to  capability  gaps,  compare  alternatives,  and  provide  information  to 
either  keep  the  baseline  system  or  procure  another  alternative. 

•  The  purpose  of  an  AoA  is  the  analytical  treatment  of  operational  effectiveness,  risk, 
and  cost  of  potential  materiel  solutions  to  fill  operational  gaps  and  determine  the 
operational  capability. 

AoAs  are  used  to  detennine  required  performance  levels  and  desired  attributes  through 
cost,  schedule,  and  performance  trades. 

•  The  AoA  also  is  used  to  determine  the  appropriate  levels  of  perfonnance  that  will  be 
required  by  the  key  perfonnance  parameters  (KPPs)  and  key  system  attributes 
through  the  tradeoff  analysis.  The  trades  are  within  and  across  alternatives  among 
cost,  schedule,  and  perfonnance.  The  trades  would  help  detennine  the  minimum 
perfonnance  requirements  needed  based  on  the  available  funding  and  schedule. 

•  An  AoA  should  help  detennine  what  the  requirements  should  be. 

•  At  MS  A,  the  AoA  is  exploring  alternatives  with  the  Initial  Capabilities  Document 
(ICD)  in  hand.  At  this  point,  the  Anny  is  not  sure  what  are  the  requirements  or  the 
desired  attributes  for  the  materiel  solution.  MS  A  AoAs  should  be  platfonn  agnostic. 

AoAs  are  comprised  of  cost,  schedule,  and  perfonnance  analyses  on  potential  materiel 
solutions. 

•  AoAs  are  comprised  of  analyses  of  materiel  solutions  to  mitigate  capability  gaps.  The 
analyses  include  cost,  schedule,  and  performance. 

•  AoAs  are  used  to  estimate  the  perfonnance  of  the  alternatives.  The  operational 
capability  of  the  alternatives  should  be  estimated  to  obtain  a  full  picture  of 
perfonnance. 

AoAs  should  support  the  acquisition  community  by  identifying  the  cost,  schedule,  and 
perfonnance  risks. 

•  The  purpose  of  an  AoA  is  to  reduce  the  risk  of  cost  and  schedule  breaches. 

•  An  AoA  should  detennine  if  industry  is  capable  of  producing  the  equipment  and 
identifying  the  cost,  schedule,  and  performance  risks. 
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AoAs  should  be  unbiased  with  no  preferred  solution,  informed  about  the  funding 
available,  and  aligned  to  national  security,  defense,  and  military  strategies. 

•  The  AoA  should  be  unbiased  with  no  preferred  solution. 

•  AoAs  must  be  resource-informed  and  be  aligned  to  national  security,  defense,  and 
military  strategies.  If  requirements  were  written  too  long  ago,  they  may  not  align  with 
these  strategies.  Changes  in  the  national  strategies  have  analytical  relevance. 

Analyses  should  be  performed  before  the  MS  A  AoA  to  identify  capability  gaps.  The 
AoA  should  start  as  soon  as  a  materiel  solution  is  identified. 

•  AoA  should  be  done  earlier  (once  the  JCIDS  process  detennines  a  materiel  solution  is 
required)  and  repeated  later  on  if  needed. 

•  There  should  be  some  efforts  before  the  MS  A  AoA,  including  identification  of  gaps 
and  development  of  the  ICD.  This  will  determine  the  emerging  set  of  requirements. 
The  preliminary  work  should  explain  why  there  is  a  legitimate  need  for  a  materiel 
solution. 

•  The  CDD  should  not  be  written  before  this  MS  A  AoA  work  is  completed.  At  MS  B, 
there  is  still  some  trade  work,  but  not  as  much  exploring  as  MS  B. 

2.  What  do  you  think  the  major  issues  are  associated  with  AoAs  under  the  new  WSARA 
guidance? 

WSARA  has  had  a  positive  impact  on  AoAs  by  increasing  the  importance  of  analysis  in 
MS  decisions.  Under  WSARA,  AoAs  have  to  be  more  accurate,  detailed,  and  unbiased.  AoAs 
can  better  support  the  requirements  development  process. 

•  WSARA  has  increased  the  value  of  AoAs.  In  the  past,  the  Services  may  have  a 
preferred  alternative,  and  the  study  may  have  been  biased  to  get  the  desired  answer. 

•  WSARA  has  been  good  for  AoAs.  WSARA  has  improved  the  MS  A  AoAs,  which 
will  provide  input  into  the  development  of  the  CDD. 

•  WSARA  has  forced  everyone  to  be  more  accurate  in  their  AoAs. 

•  There  is  also  healthy  friction  caused  by  WSARA.  The  JS  has  a  role  with  developing 
the  Source  Selection  Authority  (SSA)  and  also  owns  the  Joint  Urgent  Operational 
Needs  Statement  (JUONS)  process.  Since  the  JUONS  is  an  urgent  need,  the  JS  wants 
to  start  “bending  metal”  as  soon  as  possible.  OSD  CAPE  will  ask  where  is  the 
analysis  to  back  up  this  decision  to  start  development. 
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•  Another  issue  involves  the  growing  pains  associated  with  performing  more  detailed 
AoAs  with  increased  congressional  oversight. 

WSARA  has  increased  OSD  CAPE’s  influence  in  the  AoA  process.  OSD  CAPE  provides 
the  AoA  guidance  to  the  Services  and  determines  if  the  AoA  has  been  completed  per  its 
guidance.  WSARA  also  requires  the  DAE  to  certify  that  the  AoA  has  been  completed  in 
accordance  with  the  OSD  CAPE  guidance.  WSARA  also  provided  a  political  appointee  to  run 
OSD  CAPE. 

•  WSARA  has  increased  OSD  CAPE’s  influence.  OSD  CAPE’s  influence  over  the 
years  has  increased  and  decreased,  depending  on  administrations. 

•  Under  WSARA,  OSD  CAPE  garnered  a  lot  more  power  in  the  AoA  arena.  AoAs  are 
OSD  CAPE’s  mechanism  to  reach  down  into  the  Services  to  influence  acquisition 
decisions. 

•  OSD  CAPE  has  two  avenues  to  influence  the  acquisition  process,  including  the  AoA 
and  Program  Review  process.  The  AoA  process  is  proactive,  while  the  Program 
Review  process  is  reactive.  (Since  the  Program  Objective  Memoranda  (POMs)  are 
developed  by  the  Services,  OSD  CAPE  is  a  reviewer  of  the  program.) 

•  WSARA  has  changed  the  responsibilities  at  OSD.  OSD  CAPE  is  the  lead  for 
developing  the  guidance,  which  was  the  responsibility  of  USD(AT&L).  OSD  CAPE 
is  directly  responsible  for  AoAs.  If  an  AoA  is  deficient,  it  is  OSD  CAPE’s 
responsibility.  OSD  CAPE  determines  if  the  AoA  sufficiently  meets  the  guidance. 

•  WSARA  gave  OSD  CAPE  a  political  appointee. 

•  There  also  are  Anny  tensions  with  the  new  organization  (OSD  CAPE).  The  Army  is 
tasked  to  perfonn  the  AoA,  but  OSD  CAPE  is  responsible  for  the  AoA  being 
completed  per  their  guidance.  The  Army  should  accept  OSD  CAPE’s  new 
responsibilities. 

•  There  is  a  lot  more  scrutiny  from  OSD  CAPE.  OSD  CAPE  questions  whether  an  AoA 
is  robust  enough.  OSD  CAPE  produces  the  guidance,  but  it  is  not  specific  enough. 
OSD  CAPE  tends  to  ask  for  more  analysis  than  has  been  done  in  traditional  AoAs. 

•  WSARA  requires  AoAs  to  be  certified  by  OSD,  which  gives  OSD  leverage  over  the 
Services.  OSD  detennines  what  must  be  done  in  the  study,  and  issues  the  guidance. 
OSD  certifies  that  the  AoA  was  completed  in  accordance  with  the  guidance.  The 
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Services  run  the  risk  of  the  AoA  not  being  certified  if  it  does  not  follow  the  guidance. 
OSD  has  certified  all  AoAs  to  this  point.  OSD  CAPE/OSD  has  more  power  after 
WSARA. 

WSARA  has  also  caused  friction  between  different  organizations  in  OSD,  including 
USD(AT&L),  OSD  CAPE,  Policy  and  the  JS. 

•  There  are  many  different  tribes  in  OSD,  including  OSD  CAPE,  Policy,  and 
USD(AT&L).  There  is  some  conflict  among  them. 

•  Perhaps  an  unintended  outcome  of  WSARA  has  been  the  development  of  a 
competition  between  the  JS  that  approves  requirements  and  OSD  CAPE,  which  does 
not  believe  requirements  are  firm. 

•  WSARA  has  caused  friction  between  principal  organizations  in  OSD  including  USD 
(AT&L),  OSD  CAPE,  JS,  and  Policy.  Much  of  this  friction  revolves  around  the  SSA, 
which  is  developed  by  OSD  CAPE,  the  JS,  and  OSD  Policy.  OSD  Policy  ultimately 
approves  the  SSA.  There  is  unhealthy  fighting  where  organizations  cannot  agree  on 
the  fundamental  scenarios.  The  Services  use  scenarios  directly  from  the  SSA,  but 
OSD  CAPE  does  not  agree  with  those  scenarios.  WSARA  empowers  OSD  CAPE  in 
the  AoA  arena.  OSD  CAPE  may  not  approve  of  the  scenarios,  but  it  doesn’t  have 
alternate  scenarios. 

•  There  is  an  issue  with  the  schedule.  In  some  cases,  the  DAE  has  designated  an  X-year 
schedule  in  the  ADM,  while  OSD  CAPE  wants  to  vary  the  schedule  per  their 
guidance. 

•  WSARA  has  create  a  tension  between  USD(AT&L)  and  OSD  CAPE. 

Under  WSARA,  tradeoff  analyses  for  cost,  schedule,  and  performance  are  required  across 
and  within  alternatives.  The  trades  are  at  the  system  and  subsystem  levels,  and  they  are 
conducted  throughout  the  AoA.  WSARA  has  dictated  cost,  schedule,  and  performance  trades 
across  and  within  the  alternatives. 

•  The  tradeoff  assessments  for  cost,  schedule,  and  technology.  OSD  CAPE  has 
reinforced  these  trades  in  its  guidance  and  meetings  with  the  Anny.  Cost,  schedule, 
and  technology  combine  to  produce  a  level  of  perfonnance.  All  three  of  these  can  be 
traded  to  achieve  different  levels  of  performance  for  threshold  requirements.  The 
question  is  how  to  dial  all  three  of  these  factors  to  meet  threshold  requirements. 
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•  WSARA  requires  a  tradeoff  analysis  where  the  cost  and  performance  parameters  are 
varied  to  determine  the  resulting  program.  Schedule  also  is  to  be  considered.  There  is 
a  desire  from  OSD  CAPE  to  address  everything  at  once.  For  the  tradeoff  analysis,  one 
variable  is  held  dependent,  while  the  other  two  are  the  independent  variable. 

•  WSARA  increased  the  usefulness  of  the  AoA  by  requiring  more  trades — both  at  the 
system  and  subsystem  levels.  In  the  past,  cost  and  schedule  would  be  considered  at 
the  end  of  an  AoA.  These  cost  estimates  would  be  made  on  well-defined  alternatives. 
Under  WSARA,  the  performance  may  be  traded  for  cost  and  schedule.  This  would 
require  the  initial  cost  analysis  to  be  completed  upfront,  and  unaffordable  systems 
would  be  identified  earlier.  Both  the  cost  analysis  and  opcrational/cffcctivcness 
analysis  would  be  iterative  processes.  Ideally,  unaffordable  systems  would  no  longer 
be  initiated  only  to  get  killed  in  future  years.  By  requiring  more  trades,  the  AoA  now 
is  more  time-consuming  and  resource-intensive  than  in  the  past. 

WSARA  requires  technical/perfonnance,  cost,  and  schedule  risk  analyses.  The  risk 
analysis  is  associated  with  the  cost,  schedule,  and  performance  tradeoff  analyses. 

•  WSARA  requires  risk  analysis  (technical,  cost,  and  schedule  risk),  which  is  new. 
Tradeoff  analysis  is  mixed  into  the  perfonnance  risk.  Perfonnance  risk  can  be  thought 
of  as  the  probability  that  the  perfonnance  of  an  alternative  meets  the  requirements. 
Perfonnance  vs.  cost  can  be  examined.  The  technical  risk  includes  integration  risk 
and  manufacturing  risk. 

•  WSARA  specifies  analysis  of  trades  across  cost,  schedule,  and  perfonnance  to  reduce 
risk — these  trades  are  a  new  area  for  AoAs.  The  key  to  reducing  these  risks  is  to 
understand  the  relationships  between  cost,  schedule,  and  perfonnance.  For  example, 
to  reduce  high  schedule  risk,  the  performance  requirement  could  be  relaxed. 
Reducing  the  perfonnance  requirement  may  lessen  the  need  for  less  mature 
technology,  and  more  mature  technology  may  be  used.  Programs  have  to  reduce  the 
risk  across  the  three  areas. 

•  WSARA  has  started  the  Anny  (led  by  AMSAA)  to  develop  robust  risk  assessment 
methodology. 

•  WSARA  has  increased  the  emphasis  on  cost,  schedule,  and  performance  risk  and  the 
trades  to  mitigate  those  risks. 
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•  WSARA  also  requires  risk  analysis.  Both  the  DAE  and  OSD  CAPE  guidance  require 
risk  analysis,  which  has  not  been  standardized.  For  TRL  assessment,  there  is  no  joint 
or  defense  standard.  No  one  has  quantified  manufacturing  and  integration  risk.  There 
may  be  a  difference  in  a  OSD  CAPE  view  of  schedule  or  TRL  rating  than  the  DAE 
view. 

•  WSARA  also  mandates  that  cost,  schedule,  and  performance  risk  be  assessed.  Before 
WSARA,  the  emphasis  was  on  perfonnance  risk.  After  WSARA,  the  risks  should  be 
more  integrated,  which  is  an  improvement  on  focusing  just  on  perfonnance  risk. 

WSARA  has  increased  the  emphasis  on  cost  and  schedule  analyses  in  AoAs. 

•  The  focus  has  shifted  to  cost  and  schedule  for  AoAs.  There  is  less  emphasis  on 
perfonnance.  The  challenge  is  to  bring  cost  and  schedule  analyses  up  to  the  level  of 
the  performance  analyses. 

•  Cost  analyses  also  can  be  more  independent. 

•  WSARA  increased  the  emphasis  of  cost  earlier  in  the  acquisition  process.  Before 
WSARA,  detailed  cost  analyses  were  perfonned  for  MS  B.  WSARA  requires  detailed 
cost  estimates  for  MS  A.  To  estimate  costs  for  MS  A,  the  alternatives  need  to  be 
defined  earlier.  The  analytical  rigor  associated  with  MS  A  cost  estimates  has 
increased.  The  data  and  framework  are  not  available  at  MS  A  to  do  LCC  estimates. 

•  With  the  AoA  considering  schedule,  there  is  more  independent  control  over  the 
schedule  analysis.  For  ongoing  AoAs,  the  Anny  is  considering  options  for  the 
schedule —  although  the  PMs  don’t  want  to  provide  more  than  one  schedule. 

Under  WSARA,  the  scope  of  AoAs  has  increased.  Cost,  schedule,  and  performance  must 
be  estimated,  traded,  and  assessed  for  risk.  The  tradeoff  analyses  and  OSD  CAPE  guidance  have 
increased  the  number  of  alternatives. 

•  AoAs  under  WSARA  have  created  the  desire  to  look  at  everything,  including  cost, 
schedule,  and  perfonnance.  The  OSD  CAPE  guidance  varies  for  each  of  those 
analyses. 

•  The  OSD  CAPE  guidance  includes  many  different  factors,  and,  under  WSARA,  the 
Anny  has  to  follow  the  guidance.  The  guidance  may  dictate  a  MS  B-like  AoA  to  be 
completed  for  MS  A,  or  a  MS  A-like  AoA  for  a  MS  B  if  OSD  CAPE  wants  to  fix  the 
MS  A  AoA.  This  changes  a  linear  AoA  process  into  a  circular  one. 
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•  The  Study  Plans  seem  to  evolve  over  time  vs.  getting  finalized  early  in  the  process. 
Additional  alternatives  are  often  added  in  the  middle  of  the  study.  With  the  cost, 
schedule,  and  performance  trades,  the  system’s  requirements  will  be  evolving 
throughout  the  AoA.  These  changes  in  requirements  may  produce  additional 
alternatives  later  on  in  the  AoA.  Additional  analysis  would  have  to  be  performed  on 
these  new  alternatives. 

•  A  problem  is  the  OSD  CAPE  AoA  guidance  is  not  sufficient,  which  may  indicate 
there  is  no  common  understanding  about  AoAs.  The  instructions  in  the  guidance  are 
open  to  interpretation. 

Because  of  the  increased  complexity  and  interest  in  AoA  from  WSARA,  the  Anny  has 
experienced  an  increased  workload  in  the  analytical  community.  This  increased  workload 
requires  additional  analysts  (cost  and  performance)  and  funding  to  support  conducting  AoAs. 

•  WSARA  has  increased  the  workload  to  perform  AoAs.  There  is  greater  interest  in 
AoAs.  Because  of  this  increased  workload,  the  analysis  community  is  not  resourced 
to  perfonn  AoAs.  Funding  previously  used  to  fund  AoA  has  been  reduced  or 
eliminated. 

•  AoAs  are  not  a  lot  more  complex,  whether  it  makes  sense  to  add  this  complexity  or 
not.  OSD  CAPE  is  more  broadly  focused.  Sometimes  the  OSD  CAPE  guidance 
contains  “red  herring”  questions. 

•  WSARA  has  tripled  the  analytical  workload,  but  the  “bang  for  the  buck”  may  not  be 
totally  there. 

•  The  Number  One  issue  is  resources.  The  cost  analysis  community  did  not  increase 
personnel  levels  required  to  accomplish  the  additional  analysis  required  by  WSARA. 
Before  WSARA,  only  a  few  (four)  alternatives  would  require  cost  estimates.  After 
WSARA,  many  more  cost  estimates  are  required.  There  may  be  20  alternatives 
considered  in  the  early  screening  process,  which  may  require  some  level  of  cost 
estimates.  After  the  screening  analysis,  the  number  of  alternatives  may  drop  to  six  or 
eight.  These  alternatives  would  require  a  more  detailed  cost  estimate.  WSARA  has 
increased  the  trades  for  cost,  schedule,  and  perfonnance  for  the  subsystems  within 
alternatives.  Cost  estimates  would  be  required  for  any  of  these  trades.  With  the 
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additional  screening,  number  of  alternatives,  and  subsystem  trades,  there  is  an 
increase  in  the  required  cost  analyses. 

•  A  number  of  studies  have  concluded  that  additional  cost  analysts  were  required  in 
DoD.  The  Anny  sponsored  a  study  that  resulted  in  a  recommendation  to  add  30 
additional  cost  analysts  at  TRADOC  in  the  ARCIC  and  in  the  COEs.  The  Secretary  of 
the  Army  commissioned  a  study  of  the  Army’s  acquisition  system  by  an  independent 
panel.  The  resulting  report,  “Anny  Strong:  Equipped,  Trained  and  Ready  Final 
Report  of  the  2010  Anny  Acquisition  Review”  had  similar  conclusions.  In  response 
to  a  congressional  staff  member  request  concerning  the  shortage  of  cost  analysts, 
pricing  analysts,  and  contract  specialists,  USD(AT&L)  sent  out  a  data  call.  They 
requested  an  additional  160  cost  analysts  throughout  the  Army.  Some  of  the  analysts 
were  for  DASA-CE  and  TRADOC  WSMR  to  perform  the  additional  AoA  required 
by  WSARA. 

•  The  time  required  to  complete  the  AoA  has  increased  due  to  the  evolving  Study  Plan 
and  the  extensive  focus  on  trades. 

3,  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

AoAs  should  be  tailored  specific  to  the  program,  the  decision  space,  and  the  different  MS 
decisions. 

•  AoAs  should  be  tailored  for  different  MS,  and  how  they  are  tailored  depends  on  the 
specifics  to  the  program. 

•  There  are  similar  analyses  at  the  MSs.  The  exact  types  of  analyses  depend  on  the 
solution  set  (i.e.,  the  types  of  alternatives  drive  the  analyses). 

•  The  AoAs  should  be  tailored  for  each  MS. 

•  No  two  AoAs  are  alike. 

•  There  are  no  cookie-cutter  solutions  for  AoAs  in  MS  A  or  MS  B.  AoAs  have  to  be 
tailored  to  the  decision  space.  The  framing  of  an  AoA  will  depend  on  the  decision 
space. 

More  infonnation  and  data  are  available  for  the  later  MS  AoAs  as  opposed  to  the  MS  A 
AoAs.  The  resolution  of  the  alternatives  is  greater,  and  there  is  more  knowledge  of  the 
perfonnance. 


60 


•  The  Defense  Acquisition  Boards  (DABs)  have  different  infonnation  available  at  the 
different  MSs  with  greater  resolution  of  the  alternatives  and  analysis  from  MS  A,  MS 
B  through  MS  C. 

•  At  the  different  MSs,  there  is  a  different  playing  field  for  the  alternatives.  There  is 
much  more  knowledge  of  perfonnance  at  the  later  MSs.  The  caveats  and  limitations 
of  the  alternatives  must  be  noted.  If  data  are  available  for  an  alternative,  the  best 
representation  of  that  alternative  can  be  made.  The  paper  system  with  surrogated  data 
typically  does  better  than  actual  systems. 

•  The  level  of  detail  needed  at  MDD,  MS  A,  MS  B,  and  MS  C  is  different.  There  are 
fewer  details  available  at  MS  A.  At  MS  B,  the  cost  estimates  are  refined,  and  the 
alternatives  are  defined  in  more  detail. 

•  The  knowledge  base  is  much  different  at  MS  A  vs.  MS  B.  At  MS  B,  it  is  much  more 
mature.  The  Anny  has  to  build  knowledge  earlier  and  often  using  the  “big  tent” 
idea — bringing  in  PMs,  RDECs,  users,  etc. 

•  The  level  of  knowledge  available  is  different  between  MS  A  and  MS  B.  There  is 
much  more  knowledge  at  MS  B,  and  metal  may  have  already  been  bent.  Knowledge 
relates  to  data  and  information  used  in  the  performance  analysis.  At  MS  B,  the 
Computer-Aided  Design  (CAD)  data  are  better. 

•  The  alternatives  will  be  better  defined  at  later  MSs,  and  the  work  breakdown  structure 
has  more  detail. 

As  a  system  progresses  through  MS  A,  MS  B,  and  MS  C  decisions,  the  program  (i.e., 
budgets  and  timelines)  become  more  concrete  and  constrained.  There  are  more  tradeoffs  between 
cost,  schedule,  and  performance  at  MS  A. 

•  Also,  as  a  system  goes  from  MS  A,  MS  B,  and  MS  C,  the  program  becomes  more 
concrete  and  constrained.  The  budgets  and  timelines  become  more  constrained 
throughout  the  development  cycle.  More  tradeoffs  are  available  for  MS  A  than  MS  B 
and  MS  C.  For  the  tradeoff  analysis  at  MS  A,  all  three  factors  can  be  traded  (cost, 
schedule,  and  technology).  For  MS  B,  the  emphasis  of  the  trades  is  more  in  the 
technology  arena. 

•  Early  on  (at  MS  A)  there  are  more  assumptions,  and  it  is  more  difficult  to  play 
alternatives  in  combat  models. 
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•  AoA  are  tailored  based  on  MS,  however,  both  the  level  of  analysis  and  effort  seem  to 
be  less  for  each  MS.  For  the  AoA,  the  amount  of  work  is  probably  much  greater  for 
the  MS  A  vs.  a  MS  C.  This  difference  in  work  would  be  attributed  to  the  trades  within 
the  alternatives  in  MS  A. 

There  are  more  alternatives  and  courses  of  action  at  MS  A  than  MS  B.  MS  A  AoA  should 
consider  a  wide  range  of  technologies  and  systems. 

•  There  are  more  alternatives  early  on. 

•  The  scope  of  alternatives  and  courses  of  action  are  broader  at  MS  A  than  MS  B. 

•  The  number  of  Candida tes/alternatives  should  be  getting  smaller  as  the  program 
progresses  through  the  MSs. 

•  For  MS  A,  the  AoA  should  bracket  the  types  of  systems  and  technology  that  ought  to 
be  considered. 

•  MS  A  AoAs  should  develop  the  needed  initial  capability.  To  do  this,  the  AoA  should 
detennine  if  there  is  a  gap  and  how  to  address  the  gap.  A  breadth  of  technologies 
should  be  assessed.  A  wide  aperture  of  new  systems  should  be  considered.  More 
candidate  systems  or  technologies  need  to  be  considered  and  then  screened  down 
through  the  AoA. 

The  detail,  fidelity,  and  robustness  of  analyses  in  AoAs  increase  from  MS  A  to  MS  B  and 
MS  C.  More  detailed  scenarios  are  used  as  the  program  progresses. 

•  More  detailed  analysis  should  be  conducted  as  the  program  progresses  through  the 
MSs.  The  fidelity  of  the  analysis  also  should  increase. 

•  This  increase  of  data  and  infonnation  from  MS  A  to  MS  B  provides  a  higher  fidelity 
of  analysis;  all  analysis  is  better  at  MS  B. 

•  The  cost  estimates  are  more  robust  at  the  later  MSs,  but  the  estimation  method  is  the 
same. 

•  For  most  cost  analysis  products,  a  more  detailed  estimate  is  developed  the  system 
progress  from  supporting  a  MS  A  to  MS  B  to  MS  C.  The  tools  and  approaches  are 
different  for  developing  cost  estimates  for  these  different  MSs.  At  MS  B  and  MS  C, 
the  alternatives  would  be  better  defined  and  a  more  detailed  analysis  would  be 
perfonned.  The  quality  and  fidelity  of  the  cost  estimates  improve  as  a  system 
progresses  through  MS  B  and  MS  C. 
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•  More  detailed  scenarios  should  be  used  as  the  program  progresses.  There  should  be 
fewer  scenarios  considered  at  MS  A,  and  more  scenarios  added  as  the  program 
progresses. 

At  MS  A,  the  AoA  should  include  a  thorough  tradeoff  analysis  of  cost,  schedule,  and 
perfonnance.  The  MS  A  AoA  should  model  a  set  of  attributes  as  an  alternative,  and  the 
alternative  should  not  be  associated  with  a  system.  At  MS  A,  alternatives  can  serve  as  vessels  to 
hold  the  attributes  together  to  better  understand  the  interdependencies  of  attributes.  Perfonnance 
and  cost  estimates  will  have  to  be  made  quickly  for  the  evolving  alternatives.  First-order 
estimates  of  perfonnance  and  cost  will  have  to  be  made. 

•  At  MS  A,  there  should  be  a  very  thorough  cost-infonned  trades  effort  and  risk 
assessment.  The  MS  A  AoA  should  model  a  set  of  attributes  and  should  be  platfonn 
agnostic.  Trades  have  to  be  made  on  the  attributes  within  the  Army,  or  including 
industry.  Cost-infonned  trades  should  be  conducted  to  detennine  target  concept 
design.  Risks  should  be  assessed  for  cost,  schedule,  and  technology.  Trades  should  be 
perfonned  to  detennine  a  mature  concept  design. 

•  For  a  MS  A,  the  AoA  should  consider  all  viable  alternatives  and  then  perfonn  some 
trades  within  each  alternative. 

•  For  the  concept  design,  the  Anny  needs  first-order  estimates,  not  estimates  to  six 
significant  figures.  There  is  no  way  to  create  a  concept  design  out  of  general 
requirements  at  this  point  (MS  A  AoA).  The  concept  designs  have  to  be  developed 
rapidly  without  the  detailed  design.  We  can’t  afford  to  go  through  the  SLAD  detailed 
vulnerability  analyses,  for  example.  Complex  detailed  designs  won’t  work  for  future 
AoAs.  The  concept  designs  have  to  be  generated  quickly.  Perfonnance  and  cost 
estimates  will  have  to  be  made  for  the  evolving  concept  designs.  Because  the  concept 
design  is  evolving  due  to  trades,  these  designs  will  have  to  be  developed  quickly. 

•  For  MS  A,  the  AoA  is  focused  on  desired  attributes.  These  attributes  are  fit  together 
into  alternatives.  At  MS  A,  alternatives  can  serve  as  vessels  to  hold  the  attributes 
together  to  better  understand  the  interdependencies  of  attributes.  The  TD  phase 
includes  a  new  start  alternative  along  with  other  alternatives  (modernization,  NDI, 
etc.). 
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•  For  MS  A,  early  costing  tools  are  used  to  estimate  costs  based  on  capability.  These 
tools  are  different  than  other  cost  estimates  that  are  based  on  specific  systems  with 
known  technical  and  perfonnance  characteristics.  The  early  cost  estimating  considers 
historical  data. 

The  tradeoff  analyses  continue  at  the  beginning  of  the  MS  B  AoA,  but  later  in  the  AoA 
the  alternatives  should  be  solidified.  The  MS  B  AoA  would  compare  the  alternatives  to  the 
requirements  and  to  each  other.  The  MS  B  AoA  should  consider  broader  acquisition  options  to 
fulfill  the  needs. 

•  Work  should  continue  on  the  trades  early  in  the  MS  B  AoA  process.  Industry  should 
provide  input  on  what  can  be  done.  At  this  point,  industry  is  trying  to  build/design  to 
a  system,  and  this  work  should  be  leveraged.  The  alternatives  should  not  be  locked 
down  early  in  the  MS  B  process.  At  a  certain  point,  the  alternatives  are  solidified.  At 
this  point,  the  MS  B  AoA  would  resemble  a  classic  AoA  (i.e.,  the  AoA  would 
compare  the  alternatives  to  the  requirement  and  to  each  other). 

•  MS  B  is  the  big  decision — do  we  invest  in  a  new  program?  As  such,  the  aperture 
should  be  opened  more  to  better  understand  the  possible  options  to  fulfill  the  needs. 
The  AoA  should  talk  broader  acquisition  options  for  MS  B. 

•  MS  B  AoAs  should  finalize  the  needed  capability.  There  should  be  a  reduced  set  of 
alternatives  for  this  AoA.  The  AoA  should  ask  how  else  can  we  meet  the  needed 
capability  and  what  resources  are  available.  Options  may  include  buying  off  the  shelf 
or  new  start  program.  At  the  MS  B  AoA,  the  candidates  are  to  meet  the  capability 
gap,  and  they  should  be  compared  to  the  requirements.  Cost-effectiveness  is  a 
significant  part  of  MS  B.  Cost  and  operational  effectiveness  drivers  should  be 
identified. 

•  For  MS  B,  there  is  a  specific  set  of  alternatives  to  assess.  The  AoA  should  detennine 
if  the  new  start  alternative  is  better  than  the  baseline  alternative.  The  other 
alternatives  also  should  be  compared  to  the  baseline. 

•  For  MS  B  the  alternative  may  have  been  decided  and  the  analysis  would  focus  on 
refining  the  KPPs/Key  System  Attributes  (KSAs)  based  on  technical  feasibility, 
changing  requirements  or  cost.  For  MS  C,  again  the  AOA  would  focus  on  analysis 
around  a  given  alternative  and  further  refine  the  cost/performance  trades.  For  MS  B 
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and  MS  C,  new  alternatives  would  get  introduced  only  if  they  are  off  the  shelf  and 
could  compete  on  a  (cost/schedule/performance)  basis  with  the  current  base  case 
alternative. 

•  At  MS  B,  there  are  different  choices,  because  the  Army  is  closer  to  spending  the  big 
dollars. 

•  It  is  hoped  that  the  Service  is  smarter  going  into  MS  B. 

Although  WSARA  does  not  require  a  new  AoA  for  each  MS  decision,  conducting  AoAs 
for  each  MS  decision  would  be  beneficial.  Having  fonnal  AoAs  at  the  later  MSs  would  increase 
the  fidelity  and  structure  of  the  analysis  and  would  increase  the  information  flow  for  the 
analyses. 

•  The  idea  of  just  one  AoA  for  an  entire  program  is  not  good. 

•  AoAs  should  be  conducted  at  each  MS  (A,  B,  C  and  for  full  rate  production).  The 
AoAs  can  also  have  updates.  There  should  not  be  more  than  one  AoA  per  year  for  a 
program.  This  approach  would  keep  the  analytical  fires  burning.  If  there  is  not  an 
AoA  at  the  later  MSs,  the  analysis  is  ad  hoc  and  has  less  fidelity.  Also,  oversight  is 
lacking  without  a  fonnal  AoA.  Getting  information  to  perform  the  analysis  outside  of 
a  fonnal  AoA  won’t  be  as  easy.  AoAs  provide  a  forcing  function  to  get  information  to 
flow. 

•  To  date,  OSD  CAPE  produces  a  boilerplate  set  of  guidance  that  is  applicable  to  either 
MS  A  or  MS  B.  The  guidance  needs  to  be  sufficient  to  address  the  questions/issues 
with  a  focus  on  tipping  point  issues. 

•  WSARA  doesn’t  require  a  new  AoA  for  each  MS. 

•  Many  things  change  throughout  the  development  of  a  system  that  would  require  that 
the  AoAs  be  revisited.  Significant  changes  in  the  cost,  schedule,  or  technology  would 
constitute  a  need  for  another  AoA.  Currently,  there  is  no  policy  in  place  to  kill  a 
system  once  an  ADM  is  signed. 

•  AoA  should  be  done  earlier  (once  the  JCIDS  process  determines  a  materiel  solution  is 
required)  and  repeated  later  on  if  needed.  The  later  AoAs  would  have  more  details. 

It  would  be  beneficial  to  obtain  input  from  industry  for  MS  A. 

•  Usually,  there  is  no  input  from  industry  for  MS  A  AoAs.  There  is  a  potential  of 
contracting  with  industry  to  obtain  information,  but  we  may  not  want  to  do  that.  How 
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much  useful  information  would  we  obtain  from  industry  if  it  were  not  under  contract? 
There  should  be  more  industry  involvement. 

AoA  analysts  must  have  a  better  understanding  of  their  customers,  stakeholders,  and 
external  operating  environment. 

•  The  analysts  conducting  the  AoA  must  have  a  better  understanding  of  the 
external/operating  environment.  They  must  understand  the  stakeholders,  including 
OSD  leaders,  Army  leaders,  industry,  and  political.  They  must  be  able  to  make  the 
tough  decisions. 

•  For  a  recent  AoA  briefing  to  OSD  and  Anny  leadership,  only  three  slides  were 
reviewed  by  these  senior  leaders.  These  three  slides  listed  the  perfonnance  and  the 
cost  for  that  performance.  The  senior  leaders  were  interested  in  making  trades  rather 
than  the  details  of  the  perfonnance. 

4.  What  types  of  analysis  do  you  think  need  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 

Tradeoff  analyses  between  cost,  schedule,  and  technology/performance  should  be 
performed  throughout  the  AoA.  The  trade  space  must  be  fully  explored. 

•  Tradeoff  analyses  between  cost,  schedule,  and  technical  are  performed  to  determine 
the  effect  on  performance.  This  includes  perfonnance  at  the  system  level  and  at  the 
force  level.  Trades  should  be  conducted  through  the  study.  Both  MS  A  and  MS  B  do 
these  trades. 

•  The  analysis  must  fully  explore  the  trade  space.  The  items  open  for  trades  should  be 
identified. 

•  Cost-infonned  trades  should  be  part  of  the  AoAs 

•  Trades  and  growth  analysis  should  be  considered.  The  Anny  needs  to  make  sure 
requirements  align  with  gaps.  The  trade  space  is  large. 

•  OSD  CAPE  now  expects  cost  trades  within  alternatives  and  alternatives  that  will  be 
affordable  (alternative  constrained  by  a  cost  target).  The  trades  allow  OSD  CAPE  to 
detennine  whether  a  given  solution  may  be  good  enough  vs.  a  solution  that  meets  or 
exceeds  every  requirement.  The  trades  also  provide  OSD  CAPE  an  analytical  basis  to 
question  the  underlying  requirements. 

Risk  assessments  should  be  conducted  for  cost,  schedule,  and  technical. 
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•  Risk  assessment  should  be  conducted  for  cost,  schedule,  and  technical  (including 
integration).  Risk  analyses  are  done  at  MS  A  and  MS  B,  but  in  more  detail  for  MS  A. 

AoAs  must  address  cost  vs.  benefits,  and  cost  vs.  performance/effectiveness. 

•  Cost  and  benefits  need  to  be  analyzed  during  an  AoA.  There  is  no  comprehensive 
analysis  to  address  cost  vs.  benefits.  AoAs  tend  to  be  focused  on  the  effectiveness  of 
the  systems  for  the  benefits. 

•  OSD  CAPE  would  like  to  have  charts  that  show  the  relationship  between  the  cost  and 
the  outcomes  of  the  operational  (force-on-force)  models.  These  charts  would  show  a 
comparison  of  the  performance/effectiveness  vs.  cost.  OSD  CAPE  also  is  interested 
in  cost  of  subsystems  vs.  the  output  of  operational  models  with  these  included 
subsystems.  OSD  CAPE  is  interested  in  the  cost-effectiveness  of  these  subsystems, 
which  form  the  basis  for  the  subsystem  trades.  In  the  past,  the  analysis  was 
concentrated  on  the  system  and  how  the  system  performed. 

AoAs  should  consider  O&S  cost  and  LCC  in  addition  to  the  unit  costs. 

•  AoAs  also  tend  to  focus  on  the  unit  cost  of  the  systems.  COEAs  in  the  past  have 
addressed  the  other  areas  besides  effectiveness  and  unit  cost  more  thoroughly  than  the 
current  AoAs. 

•  New  start  systems  are  never  cheaper  to  maintain.  The  LCC  and  O&S  costs  need  to  be 
considered. 

AoAs  should  consider  other  aspects  besides  effectiveness,  such  as  the  following: 
reliability,  DOTMLPF,  strategic  impacts,  transportability,  maintainability,  manpower  analyses, 
force  structure  impacts,  and  squad  size  impacts. 

•  Some  aspects  may  be  more  important  than  effectiveness — such  as  reliability, 
DOTMLPF,  strategic  impacts,  transportability,  maintainability,  manpower  analyses, 
force  structure  impacts,  and  squad  size  impacts. 

AoAs  should  include  the  following  types  of  analyses:  mission  effectiveness, 
performance,  cost,  affordability,  tradeoff  (cost,  schedule,  performance/technology), 
sensitivity/parametric,  risk  (cost,  schedule,  and  technical),  growth  assessment,  sustainment,  fuel 
consumption,  energy  consumption,  DOTMLPF,  fleet  strategy,  and  portfolio. 

•  The  types  of  analyses  include  the  following:  performance,  mission  effectiveness,  cost, 
risk  assessment  (technical,  schedule  and  cost),  growth  assessment,  sustainment  and 
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others  on  a  case-by-case  basis.  The  growth  assessment  is  needed,  because  of  the 
current  cost  environment. 

•  Analysis  should  consider  the  fuel  aspect  (i.e.,  electrical  power,  fuel  or  energy 
consumption). 

•  Typical  analyses  perfonned  in  AoAs  include  mobility,  power  and  energy, 
perfonnance,  operational  effectiveness,  cost,  schedule,  and  performance  risk  analysis, 
and  trades,  etc.  Cost  is  especially  important  at  MS  B. 

•  The  AoA  must  assess  the  performance  characteristics  of  the  alternatives  and  their 
impact  on  perfonnance  and  operational  effectiveness. 

•  For  MS  A,  there  are  similar  analyses,  but  affordability  and  cost  analysis  are  not 
stressed  as  much. 

•  For  MS  B,  the  AoA  needs  to  include  performance,  operational  effectiveness, 
logistics,  sustainment,  LCC,  and  affordability  analyses.  DOTMLPF  analysis  also 
should  be  perfonned  to  help  decision-makers  understand  the  implications  of  pursuing 
a  particular  alternative. 

•  Cost-effectiveness  analysis  should  be  performed  and  goals  prioritized.  Cost  reduction 
should  be  examined. 

•  Fleet  strategy  analysis  should  be  conducted.  For  example,  one  third  of  the  fleet  would 
be  given  the  best  new  start  system  and  two-thirds  of  the  fleet  would  be  provided  a 
modernization  system. 

•  Parametric  analysis  is  important,  especially  for  MS  A.  This  analysis  will  help 
detennine  the  interrelationship  between  perfonnance/operational  effectiveness 
metrics  (e.g.,  lethality,  survivability,  and  force  protection).  For  example,  a  better 
annor  package  would  increase  survivability  and  would  allow  the  vehicle  to  survive 
longer  in  the  battle.  The  vehicle  would  be  able  to  engage  more  enemy  vehicles,  which 
would  result  in  greater  lethality.  A  larger  weapon  could  engage  the  enemy  more 
effectively,  which  would  increase  the  vehicle’s  survivability.  The  tradeoff  analysis  is 
related  to  the  parametric  analysis. 

•  WSARA  requires  cost,  schedule,  and  perfonnance  trades  to  be  completed  in  AoAs. 
DoD  5000  calls  out  LCC  estimates,  operational  effectiveness,  and  perfonnance 
analysis.  A  DOTMLPF  analysis  also  should  be  performed  to  identify  the  implications 
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of  a  particular  alternative.  The  implications  could  be  hidden  costs  for  sustainment. 
Other  items  to  consider  are  sustainment  impacts,  operational  benefits,  force  structure, 
and  DOTMLPF. 

•  After  WSARA,  the  analysis  has  expanded  to  the  subsystems  and  the  portfolio  of 
systems. 

•  The  Capability  Knowledge  Base  should  be  populated  with  ACAT  II  and  ACAT  III 
data. 

•  Item  and  system  level  performance  analysis  are  key  parts  of  the  AoA.  Force 
protection  and  mobility  are  important  performance  analyses.  Other  analyses  include 
the  following:  cost,  risk,  trades,  armor,  other  than  armor,  geospatial,  C4, 
transportability,  mobility,  power  and  energy,  Size,  Weight,  and  Power  (SWAP), 
reliability,  Class  IX,  sensors,  and  Chemical,  Biological,  Radiological,  and  Nuclear 
(CBRN). 

The  analyses  must  be  transparent,  open,  and  understandable  to  OSD  CAPE.  The  analyses 
have  to  address  OSD  CAPE’s  decision  choices  and  “tipping  points.”  Analyses  have  to  be 
objective,  robust,  and  credible. 

•  The  analysis  must  be  transparent  and  open  to  OSD  CAPE. 

•  AoAs  failed  to  speak  to  OSD  CAPE  in  their  language.  OSD  CAPE  does  not  have  the 
capability  to  do  the  AoAs.  They  are  program  evaluators.  The  Anny  has  to  learn  to 
speak  to  them  in  ways  that  are  relevant  to  their  decision  choices.  Decision  choices  are 
not  necessarily  alternatives. 

•  The  Army  must  be  much  more  transparent  when  dealing  with  OSD  CAPE.  OSD 
CAPE  should  be  invited  to  meetings  and  In-Process  Reviews  (IPRs).  The  Anny  has 
to  educate  them  early  and  often.  Although  feeding  them  infonnation  early  could 
enable  OSD  CAPE  to  use  it  against  us,  the  Army  has  opted  to  bring  them  in  on  recent 
AoAs.  The  goal  is  to  convince  OSD  CAPE  that  our  analysis  is  unbiased  and  credible. 

•  The  analyses  have  to  be  transparent  to  OSD  CAPE — OSD  CAPE  has  to  understand 
the  analysis  and  be  assured  that  their  “tipping  points”  are  being  addressed. 

•  The  analysis  must  be  objective  and  not  favor  one  alternative,  credible  and  robust,  and 
tailored  to  the  decision  space. 
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•  Past  AoAs  have  compared  the  improve  performance  of  the  new  start  vs.  base  case. 
This  analysis  doesn’t  tell  us  what  is  good  enough. 

•  The  analysis  needed  in  an  AoA  depends  on  must  be  tailored  to  the  decision. 

•  OSD  CAPE  requires  an  analytically  complete  product  regardless  of  whether  the  data 
can  support  the  analysis.  For  example,  there  may  not  be  data  to  perform  the  LCC 
estimates.  There  may  not  be  an  analogous  system  to  estimate  cost. 

OSD  CAPE  guidance  drives  the  types  of  analyses  perfonned  in  AoAs,  but  the  guidance  is 
not  consistent  across  AoAs. 

•  The  key  is  that  there  is  no  consistent  OSD  expectation  for  AoAs.  There  are  different 
AoA  expectations  depending  on  the  system,  and  the  OSD  CAPE  person  assigned  to 
the  AoA.  OSD  CAPE  personnel  don’t  perfonn  analysis,  but  rather  check  analyses  that 
the  Services  produce. 

•  The  Army  should  examine  AoAs  that  OSD  CAPE  believes  were  done  well,  and  they 
should  leam  what  analysis  was  performed. 

•  OSD  CAPE  guidance  drives  what  type  of  analysis  is  used  in  the  AoA. 

•  Special  interest  questions  may  be  asked  by  OSD — for  example,  operational  energy. 

5,  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

There  are  four  basic  types  of  alternatives  to  include  the  system  in  service,  modifications 
of  the  system  in  service,  NDI/foreign  systems,  and  new  start  programs. 

•  There  are  basically  four  general  types  of  alternatives,  including  the  following: 
currently  available  systems  (do  nothing  option);  Product  Improvement  Programs 
(PIPs)  for  existing  system  (i.e.,  modernize  the  existing  system);  off  the  shelf  (OTS) 
foreign  or  domestic  systems;  and  new  start  programs. 

•  There  are  three  categories  of  AoAs:  modifications,  new  starts,  and  NDIs. 

•  Modernized  versions  of  current  U.S.  systems  should  be  considered,  especially  early 
on. 

•  A  baseline  alternative  is  required  to  represent  what  capability  is  available  today.  The 
other  alternatives  represent  greater  capability  than  what  is  currently  available. 

•  Early  on  it  is  helpful  to  include  the  current  system,  if  available,  and  a  minimal  amount 
of  modifications  to  keep  it  operational.  That  would  be  the  lower  bounds  on  both  cost 
and  perfonnance. 
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A  screening  analysis  during  the  AoA  would  be  beneficial  in  detennining  an  initial  set  of 
alternatives.  The  screening  analyses  starts  with  a  broad  look  for  possible  alternatives  and  reduces 
these  alternatives  based  on  criteria. 

•  In  previous  AoAs,  screening  analysis  has  been  used  to  start  from  a  larger  set  of 
potential  alternatives  and  reduce  it  to  an  initial  set  of  alternatives. 

•  Screening  analysis  is  important  to  help  develop  and  refine  the  definitions  of 
alternatives.  Screening  analysis  is  necessary  to  remove  alternatives  that  do  not  make 
sense  and  to  refine  alternatives  definition  based  on  criteria  from  COE  and  Army 
leadership.  The  screening  analysis  is  led  by  the  AoA  team,  but  the  SAG  ultimately 
makes  the  decision  on  the  AoAs.  This  screening  analysis  will  help  shape  the  decision 
space. 

•  Alternatives  should  be  defined  in  a  stepwise  manner.  At  first,  a  wide  net  should  be 
cast  for  possible  alternatives.  A  focused,  disciplined  selection  process  should  be 
undertaken  to  reduce  the  set  of  alternatives  to  a  management  number. 

•  Screening  analysis  can  be  used  to  define  the  alternatives.  The  criteria  in  the  screening 
analysis  really  highlight  viable  alternatives.  The  question  is  what  alternatives  go 
forward  in  the  AoA  for  detailed  analyses. 

•  The  AoA  should  start  off  with  as  broad  a  look  as  possible  for  alternatives.  A 
screening  process  can  be  used  to  reduce  the  number  of  alternatives.  For  this  reduced 
number  of  alternatives,  a  more  detailed  analysis  will  be  conducted. 

•  The  team  working  the  study  should  define  the  alternatives  with  input  from  the 
stakeholders  and  customers.  The  Army  stakeholders  think  more  narrowly  than  the 
OSD  customers. 

•  It  would  be  beneficial  to  have  independent  organizations  define  the  alternatives. 

•  The  AoA  team  should  ensure  if  there  are  a  couple  of  requirements  that  are  knocking 
out  alternatives  early  (i.e.,  during  a  screening  analysis).  The  team  should  make  sure 
these  requirements  are  valid. 

OSD  CAPE  plays  a  significant  role  in  the  development  of  alternatives.  OSD  CAPE 
provides  the  AoA  guidance  and  chairs  the  SAG  which  approves  the  alternatives.  OSD  CAPE 
should  not  be  involved  with  the  Services’  force  design. 

•  It  would  be  beneficial  to  have  agreement  with  the  set  of  alternatives  going  into  OSD. 
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•  OSD  leadership  is  learning  what  it  would  like  to  have  in  terms  of  alternatives. 

•  OSD  CAPE  should  not  be  into  force  design.  Force  design  is  partly  an  operational  art 
and  should  be  left  to  the  Services.  AoAs  should  not  jumble  force  design  with  materiel 
solutions  or  alternatives. 

•  The  alternative  definitions  are  developed  from  OSD  CAPE.  OSD  CAPE  is  exposed  to 
ideas  and  infonnation  that  influence  its  identification  on  alternatives.  OSD  CAPE’s 
horizon  is  broader  than  the  Army.  OSD  CAPE’s  input  is  necessary  as  well  as 
alternatives  from  the  user  community  to  address  capability  gaps  it  wants  mitigated. 

•  The  OSD  CAPE  guidance  and  the  AoA  Study  Director  should  define  the  alternatives. 

The  number  of  alternatives  should  be  managed  to  a  reasonable  number  (six  or  fewer)  to 

complete  the  AoA  in  the  9-  to  12-month  timeframe.  Alternatives  should  be  eliminated  if  they  are 
detennined  to  be  infeasible,  and  there  should  be  a  limited  number  of  “special  interest”  systems. 

•  The  number  of  alternatives  has  to  be  reasonable,  because  the  typical  time  to  do  an 
AoA  is  1  year. 

•  The  number  of  alternatives  should  be  managed  to  a  reasonable  number 
(approximately  six  or  fewer).  Having  this  number  of  alternatives  should  keep  the 
AoA  duration  around  9  to  12  months. 

•  It  is  difficult  to  cut  out  alternatives.  The  alternatives  should  be  defined,  but  they  keep 
changing  throughout  the  AoA.  OSD  CAPE  keeps  adding  alternatives.  Some 
alternatives  added  are  “nonstarters,”  but  they  are  never  eliminated.  Alternatives  are 
added,  but  they  are  never  subtracted.  It  would  be  good  to  cut  some  alternatives  out. 

•  Other  organizations  may  add  “special  interest”  alternatives  that  will  continue  through 
analysis  even  though  they  do  not  necessarily  make  sense. 

•  These  alternatives  should  have  a  clear  distinction  of  capabilities.  There  should  also  be 
a  minimum  allowable  percentage  of  politically  motivated  alternatives  (i.e.,  special 
interest  cases).  The  loud  person  in  the  SAG  should  not  force  special  interest 
alternatives  into  the  AoA. 

•  Another  question  is  whether  special  interest  alternatives  continue  forward. 

•  Solutions  that  are  not  feasible  will  be  screened  out  and  should  not  be  included  in  the 
AoA. 
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Defining  alternatives  is  a  both  a  top-down  and  bottoms-up  process.  The  top-down  process 
is  based  on  guidance  from  Army  and  OSD  leadership,  and  the  bottoms-up  approach  is  based  on  a 
screening  analysis. 

•  The  process  for  detennining  alternatives  is  a  twofold  process.  A  top-down  process  is 
based  on  guidance  from  Army  and  OSD  leadership.  The  bottoms-up  approach  is 
based  on  a  screening  analysis.  The  alternative  definition  process  is  a  combination  of 
these  two  processes.  Defining  the  alternatives  is  accomplished  by  consensus  among 
Anny,  OSD,  and  AoA  Study  leadership. 

For  MS  A,  the  alternatives  should  be  attribute-based  and  developed  by  the  materiel 
developer,  user,  analyst,  and  industry. 

•  For  MS  A,  the  alternatives  should  be  attribute-based.  The  alternatives  should  be 
developed  by  the  materiel  developer,  user,  analyst,  and  industry.  The  Integrated 
Capabilities  Development  Team  (ICDT)  should  be  used  to  go  through  the  trades.  The 
alternatives  should  be  a  set  of  attributes  and  should  be  platform-agnostic. 

•  For  MS  A,  the  purpose  of  alternatives  is  to  determine  requisite  attributes. 

•  The  knowledge  at  the  time  drives  the  definition  of  alternatives.  If  only 
concepts/technologies  are  available,  the  alternatives  will  be  a  collection  of  concepts 
and  technologies  rather  than  a  full  system.  If  the  knowledge  is  more  mature,  the 
alternatives  can  be  full  systems.  Alternatives  can  be  foreign  systems. 

•  For  the  early  part  of  MS  B  AoA,  there  should  be  really  good  input  from  industry. 
Trades  should  continue.  At  some  point,  the  AoA  team  should  develop  more  defined 
altematives/concept  designs. 

•  The  acquisition  guys  perform  market  research.  There  has  to  be  communication 
between  the  PM/PEO  shops  and  the  analytical  teams. 

Alternatives  could  be  a  mix  of  different  systems  for  the  fleet.  An  alternative  could 
include  some  new  start  systems  for  a  portion  of  fleet  and  some  modified  systems  for  the 
remainder  of  the  fleet. 

•  Sometimes  alternatives  can  be  a  mix  of  different  systems.  An  alternative  could 
include  some  new  start  systems  for  a  portion  of  fleet  and  some  modified  systems  for 
the  remainder  of  the  fleet. 

There  are  more  alternatives  in  a  MS  A  AoA. 
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•  At  MS  A,  there  are  more  alternatives. 

•  There  is  a  broad  range  of  alternatives  for  AoAs 

There  were  other  points  identified  for  the  process  of  defining  alternatives. 

•  The  capability  developer  would  detennine  if  there  is  a  nonmateriel  way  of  meeting 
the  gap. 

•  The  AoA  should  consider  all  viable  alternatives  without  giving  preferential  treatment 
to  the  acquisition  or  user  community’s  preferred  alternative. 

•  With  the  addition  of  the  trade  analysis,  the  level  of  perfonnance  of  the  new  start 
alternative  may  not  be  a  great  as  in  past  AoAs.  The  cost  and  available  funding  may 
constrain  the  level  of  performance  attainable  for  a  new  start. 

•  The  alternatives  should  be  defined  to  show  differences  in  operational  values. 

•  The  alternatives  should  be  defined  as  much  as  possible  early  in  the  AoA  process, 
although  it  is  impossible  to  totally  define  them  in  the  beginning.  The  definition  of 
alternatives  starts  with  the  requirements.  As  the  AoA  progresses  and  the  requirements 
are  refined,  the  definitions  of  the  alternatives  are  better  defined. 

•  The  level  of  detail  or  description  of  an  alternative  is  somewhat  dictated  by  the  phase 
of  the  program.  The  alternatives  have  to  be  adequately  described  to  allow  both  the 
cost  estimators  and  operational/effectiveness  war-gaming  analysts  to  quantify 
differences  between  alternatives  to  allow  comparison  and  trades. 

•  It  would  be  nice  to  have  guidance/rules  of  thumb  addendum  to  WSARA  on 
alternatives.  The  process  now  is  ad  hoc. 

•  If  the  CDD  is  available,  the  threshold  and  objective  requirements  can  be  used  to  help 
define  the  alternatives.  The  requirements  can  be  traded  to  get  different  alternatives. 

6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

For  foreign  systems,  it  is  difficult  to  obtain  the  level  of  data  needed  for  AoA  performance 
and  cost  analyses.  Foreign  countries  or  companies  are  not  willing  to  share  detail  data  on  their 
systems,  because  they  don’t  believe  the  United  States  will  purchase  them.  At  some  point,  data 
might  be  available  for  one  configuration,  but  not  the  configuration  the  AoA  requires. 

•  There  is  a  lack  of  good  solid  information  about  foreign  alternatives.  There  are  limited 
data  and  the  analytical  comparison  is  not  “apples-to-apples.”  In  some  case,  the  United 
States  does  have  data  on  the  foreign  system,  but  modifications  must  be  made  to  the 
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system — which  reduces  the  value  of  the  data.  Sometimes  the  data  are  only  on  a  part 
of  the  foreign  system.  Usually,  the  foreign  data  have  low  fidelity  and  limited  detail. 
Any  test  data  from  foreign  countries  are  suspect. 

•  It  is  difficult  to  get  the  data.  In  some  cases,  the  United  States  can’t  get  the  data. 

•  The  AoA  team  would  have  to  rely  on  vendor  data  for  the  cost  estimates.  There  may 
have  to  be  site  visits  to  obtain  data. 

•  The  data  for  the  foreign  alternatives  are  not  as  available  as  U.S.  alternatives.  The  data 
for  the  foreign  system  that  may  already  be  built  may  not  be  any  better  than  the  data 
for  a  developmental  system.  There  is  an  “apples-to-oranges”  relationship  between  the 
data  for  the  foreign  systems  and  the  other  systems. 

•  There  will  be  more  of  a  problem  with  foreign  systems  at  MS  B,  because  more 
detailed  system  data  are  required.  This  may  be  less  of  a  problem  for  MS  A,  because 
we  are  comparing  attributes.  In  some  cases,  we  may  not  even  know  the  attributes  for 
the  foreign  systems.  There  is  zero  tradeoff  work  for  foreign  systems,  because  we 
don’t  know  enough  about  them. 

•  The  data  available  for  foreign  systems  are  not  deep  enough. 

•  Foreign  systems  are  already  mature,  and  it  is  not  that  difficult  to  estimate  costs, 
providing  companies  are  willing  to  share  data.  It  can  be  difficult  to  estimate  costs  of 
foreign  systems  due  to  the  lack  of  data.  If  we  can  obtain  some  cost  data  and  technical 
characteristics,  we  can  estimate  foreign  systems  costs.  It  is  good  to  have  these 
systems  included. 

•  Sometime  there  is  not  enough  infonnation  on  the  foreign  alternatives,  and  the 
surrogation  process  for  data  will  make  including  the  foreign  alternative  meaningless. 
The  data  problem  needs  to  be  solved. 

•  These  data  are  needed  to  compare  the  foreign  system  to  other  alternatives. 

•  It  is  difficult  to  get  the  other  countries  to  share  the  same  level  of  infonnation.  Because 
the  United  States  never  selects  foreign  systems,  the  countries  are  reluctant  to  share 
their  data. 

•  Potentially  no  data  available  for  the  configuration  the  United  States  wants  to 
purchase.  Surrogation  must  be  used,  but  the  surrogation  must  be  as  close  as  possible. 
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•  Foreign  alternatives  are  a  huge  problem,  especially  regarding  data.  It  is  not  in  the  best 
interest  of  the  countries  to  provide  infonnation  to  the  United  States,  especially  the 
detailed  infonnation  that  AMSAA  and  ARL  SLAD  require. 

•  There  should  be  an  appropriate  level  of  data  for  an  “apples-to-apples”  comparison  of 
the  foreign  alternative  to  the  other  U.S.  alternatives.  If  the  appropriate  level  of  detail 
is  not  available,  surrogations  would  have  to  be  made.  The  AoA  must  be  upfront  with 
the  limitations  on  data  with  the  foreign  alternatives.  There  should  be  a  judgment  of 
the  quality  of  the  foreign  system  data. 

•  From  a  cost  standpoint,  foreign  system  estimates  for  the  life  cycle  are  developed  in  a 
similar  manner  as  with  other  systems.  It  is  more  difficult  to  obtain  both  cost  and 
perfonnance  data  for  these  systems,  but  we  have  been  successful  in  the  past. 

•  Once  some  of  the  foreign  companies  detennine  that  we  may  seriously  consider  their 
systems,  I  think  they  will  be  more  willing  to  share  the  cost  and  technical  data. 

Foreign  systems  should  be  identified  and  agreed  upon  early  in  the  AoA  process.  The  data 
and  infonnation  request  process  should  start  early. 

•  Foreign  systems  should  be  identified  early  in  the  AoA  process  and  requests  for 
information  sent  out  to  obtain  data.  For  the  item-level  analysis,  detailed  data  are 
required  for  the  foreign  systems. 

•  The  Anny  should  aggressively  pursue  obtaining  as  much  data  as  possible. 

•  An  early  definition  of  foreign  systems  is  needed  as  well  as  a  determination  of  what  is 
possible  for  foreign  systems. 

The  AoA  team  should  not  have  the  responsibility  to  obtain  the  data  for  foreign  systems. 

•  The  AoA  team/analysts  should  not  have  the  burden  to  get  the  data  for  foreign 
alternatives.  There  are  too  many  foreign  and  exchange  agreements.  Maybe  OSD 
CAPE  or  USD(AT&L)  can  obtain  this  data. 

•  The  organization  that  selects  the  foreign  system  for  inclusion  in  the  AoA  needs  to 
ensure  that  the  data  can  be  acquired. 

An  Army  PM  is  required  to  develop  a  notional  schedule  for  the  foreign  alternative  as 
input  into  the  tradeoff  and  risk  analyses. 

•  There  would  have  to  be  a  PM  associated  with  the  foreign  system  to  develop  a 
notional  schedule,  which  would  include  testing. 
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All  foreign  systems  will  require  some  level  of  modification  to  address  U.S.  standards, 
U.S.  GFE  (e.g.,  radios),  or  U.S.  manufacturing  sites.  Modifications  may  also  be  required  to  meet 
desired  capability.  The  cost  and  schedule  of  these  modifications  would  have  to  be  included  in  the 
AoA. 

•  It  is  likely  that  all  foreign  systems,  if  selected,  would  have  to  be  modified.  The  only 
real  NDI  systems  would  be  U.S.  systems. 

•  Selecting  a  foreign  system  “as  is”  would  probably  involve  settling  for  less  than  what 
is  set  forth  in  the  requirements.  Modifications  to  the  foreign  systems  can  be  included 
to  increase  capability  to  meet  the  requirements.  The  cost  and  schedule  of  these 
modifications  would  have  to  be  included  in  the  AoA. 

•  The  foreign  alternatives  always  require  some  modification.  Costs  and  impacts  of  the 
modifications  must  be  accounted  for. 

•  It  should  be  noted  that  foreign  alternatives  are  not  NDI  alternatives.  All  foreign 
alternatives  will  have  to  be  modified  to  be  used  by  the  U.S.  and  to  meet  U.S. 
standards. 

•  Also,  the  foreign  systems  are  not  really  NDIs.  They  require  significant  additional 
modification. 

•  Any  foreign  system  will  have  to  be  Americanized.  Foreign  systems  can’t  just  be 
bought  off  the  shelf.  The  ammunition,  radio,  etc.,  would  have  to  be  changed  meet 
U.S.  standards.  The  system  may  have  to  be  altered  to  fit  U.S.  soldiers.  This 
modification  of  the  foreign  system  must  be  included  in  the  cost  of  the  foreign 
alternative.  Manufacturing  in  the  United  States  or  another  country  must  be  considered 
in  the  cost  analysis.  All  assumptions  need  to  be  laid  out. 

•  If  a  foreign  system  is  selected,  it  would  have  to  be  built  in  the  United  States. 

There  are  risks  associated  with  using  foreign  systems  such  as  compatibility, 
interoperability,  and  sustainment.  These  risks  must  be  considered  in  the  AoA. 

•  There  are  issues  of  compatibility  in  using  foreign  systems. 

•  The  risks  associated  with  the  foreign  alternatives  not  being  made  in  the  United  States 
must  be  considered. 

•  A  number  of  factors  not  usually  considered  in  AoAs  are  important  to  the  risk  of  using 
foreign  alternatives.  For  example,  the  interoperability  of  the  foreign  system  in  the 
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U.S.  Army,  the  fuel  the  system  uses,  recovery  vehicle  used  for  the  foreign  system,  or 
special  toolkits  required  to  repair  the  systems.  Factors  such  as  these  must  be 
considered  before  the  MS  decisions  are  made. 

The  United  States  rarely  buys  foreign  systems  because  of  political  reasons  and  industrial 
base  reasons. 

•  The  United  States  rarely  buys  the  foreign  alternative,  but  it  is  good  to  include  them. 

•  The  possibility  of  actually  buying  a  foreign  alternative  is  small,  because  of  the 
political  firestorm  it  would  cause. 

•  Another  issue  is  that  a  foreign  system  may  not  be  politically  acceptable.  The  SAG 
will  have  to  eliminate  systems  that  would  not  be  acceptable. 

•  There  is  hesitance  to  use  foreign  systems  unless  they  are  allowed  into  open 
competition,  which  may  not  be  acceptable  from  a  taxpayer’s  standpoint.  There  is  a 
need  to  build  the  system  in  the  United  States.  The  Untied  States  should  have  the 
capability  to  build  its  weapons  systems.  The  United  States  does  not  want  to  be  held 
captive  by  another  country  for  spare  parts.  There  have  been  issues  in  the  past  about 
not  getting  spare  parts  from  other  countries  where  the  State  Department  had  to  step  in 
to  help.  Another  issue  is  that  if  the  United  States  buys  a  system  from  one  country, 
other  U.S.  allies  may  be  upset  at  this  favoritism. 

•  Most  foreign  companies  selected  to  produce  systems  for  the  Army  end  up  setting  up 
facilities  in  this  country  to  appease  Congress  and  meet  buy- American  rules. 

•  It  is  questionable  whether  foreign  systems  are  feasible  alternatives — i.e.,  would  they 
actually  be  procured?  There  may  be  industrial  base  concerns  with  purchasing  a 
foreign  system.  Congress  may  not  support  procuring  a  foreign  system,  and  it  may  not 
be  politically  acceptable. 

Foreign  alternatives  should  be  considered  appropriately  based  on  the  information/data 
available.  There  may  be  greater  uncertainty  in  the  analytical  results  of  foreign  systems.  This 
uncertainty  should  be  understood  by  the  AoA  customers. 

•  The  challenge  is  how  to  make  the  foreign  system  comparable  in  terms  of  analysis. 

•  Foreign  alternatives  should  be  considered  in  the  appropriate  manner. 

•  Foreign  alternatives  should  be  considered  in  AoA  to  the  extent  possible  in  a  way 
suitable  to  the  infonnation  available  on  them.  They  may  not  be  treated  in  a  similar 
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manner  as  the  U.S.  systems,  because  of  data  available.  The  foreign  alternatives  may 
be  treated  with  greater  uncertainty. 

•  Foreign  alternatives  should  be  considered  appropriately  with  the  same  standards  as  all 
alternatives. 

•  Foreign  systems  should  be  considered  like  any  other  systems  that  require  minimal 
development  (such  as  some  NDI  systems). 

•  Dealing  with  foreign  alternatives  is  very  complicated,  but  the  Army  must  learn  how 
to  incorporate  them  into  AoAs. 

•  “Fuzzy  dots”  are  acceptable  to  OSD  CAPE,  but  they  tend  to  look  at  the  center  of  the 
dots  as  the  most  likely  point  and  use  that  as  if  it  were  quality  data. 

•  There  is  never  an  equal  understanding  of  foreign  alternatives  as  for  U.S.  alternatives, 
but  there  are  huge  expectations  for  the  analysis. 

The  number  of  foreign  alternatives  should  be  limited  (one  or  two),  because  of  the 
analytical  resources  needed  to  evaluate  them. 

•  There  should  be  only  one  or  two  foreign  systems  included  in  the  AoA. 

•  Foreign  alternatives  should  be  considered  in  the  screening  analysis. 

•  Consideration  of  foreign  alternatives  requires  analytical  resources  to  evaluate  them. 

Foreign  systems  should  not  be  considered  if  they  are  not  viable,  feasible  candidates. 

•  Foreign  systems  should  not  be  included  if  they  are  not  serous  candidates.  The  Army 
does  not  want  to  waste  analytical  resources  on  a  throwaway  system  just  to  say  a 
foreign  system  is  considered. 

•  If  a  foreign  system  is  a  viable  alternative,  it  should  be  considered 

•  If  an  alternative  is  not  feasible,  it  should  not  be  included. 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs? 

The  trades  are  between  cost,  schedule,  and  technology.  These  trades  would  result  in  a 
level  of  performance.  Tradeoff  analyses  involve  parametric  analyses  where  technologies  are 
added  or  removed  to  the  alternatives. 

•  The  trades  between  cost,  schedule,  and  technology  would  result  in  a  level  of 
perfonnance.  A  mathematical  representation  of  the  tradeoff  analysis  would  be  f(c,  s, 
t)  =  p,  where  c  =  cost,  s  =  schedule,  t=  technology,  and  p  =  perfonnance.  This  level  of 
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performance  developed  from  the  tradeoff  analysis  would  inform  an  AoA.  AoAs 
typically  address  performance  and  operational  effectiveness,  but  they  don’t  trade 
technologies.  Each  alternative  has  its  own  level  of  performance.  These  levels  of 
perfonnance  for  the  alternatives  are  compared  during  an  AoA.  A  change  in 
perfonnance  could  be  the  justification  for  doing  an  AoA  for  MS  B  or  MS  C. 

•  Trades  involve  parametric  analyses.  There  can  be  many  combinations  of  trades  by 
taking  off  or  adding  technologies  to  alternatives. 

A  systematic  approach  to  the  tradeoff  analysis  should  be  developed  and  used  for  AoAs. 

•  The  tradeoff  analysis  is  an  integral  part  of  an  AoA.  The  analysis  should  be  composed 
of  a  couple  of  steps.  Phase  1  would  be  a  parametric  analysis  and  a  capability 
comparison  (this  would  be  within  an  alternative  comparison).  Following  this,  a 
systems  trades  analysis  should  be  conducted.  There  should  be  a  systematic  approach 
to  trades  and  possibly  a  tool  concept. 

•  After  the  AoA,  the  next  MS  will  also  include  trades.  For  MS  A,  it  is  what  the  Anny 
wants.  For  MS  B,  it  is  what  the  industry  does  ;  for  MS  C,  what  can  industry  really  do. 

•  Someone  has  to  define  what  is  to  be  traded  and  compared.  The  AoA  team  needs  help 
as  to  what  to  trade. 

Affordability  analysis  should  be  conducted  to  determine  a  cost  target  range,  which  can  be 
used  in  the  tradeoff  analyses. 

•  For  tradeoff  analysis,  everything  should  not  be  varied  at  once.  There  should  be  a  cost 
target  range,  but  there  also  should  be  excursions  outside  that  range.  The  schedule 
should  be  varied  to  minimize  risk. 

•  Affordability  must  be  included  early  in  the  AoA  process  instead  of  after  the  AoA. 
From  the  affordability  analyses,  cost  targets  can  be  generated. 

•  Most  of  our  current  AoA  establish  a  procurement  cost  target  used  to  perfonn  trades 
on  alternatives  and  technologies  throughout  the  AoA  process.  Both  the  cost  analysis 
and  operational/effectiveness  analyses  are  iterative  processes.  Alternatives  that  cost 
more  than  the  target  are  excluded. 

The  tradeoffs  must  be  done  within  the  context  of  the  decision  space. 

•  The  tradeoffs  must  be  done  within  the  context  of  the  decision  space.  There  must  be  a 
balance  of  cost,  schedule,  and  perfonnance  within  the  decision  space.  In  the  past,  the 
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decision  space  was  ignored  to  preserve  the  sanctity  of  the  analysis.  The  analysis  was 
very  thorough,  but  it  may  not  have  supported  the  decision  space. 

•  The  Anny  is  forced  to  live  with  the  cost,  schedule,  and  perfonnance  trades.  The 
Anny  has  to  detennine  how  to  make  the  trades  acceptable  to  OSD  CAPE. 

Operational  effectiveness  and  performance  analyses  are  required  to  perform  the  tradeoff 
analyses. 

•  Both  item/system  level  performance  and  force-on-force  effectiveness  results  are 
needed — although  sometimes  the  trades  are  done  without  force-on-force  analyses. 

•  There  still  are  trades  in  the  AoA,  and  detailed  analysis  will  help  detennine  what  to 
trade. 

It  is  difficult  to  consider  schedule  trades  in  the  tradeoff  analysis.  The  desired  timeline  for 
development  of  the  system  may  preclude  a  change  in  the  schedule.  There  is  a  lack  of  realistic 
schedule  estimation,  and  schedule  variations  greatly  affect  cost. 

•  Trades  can  be  done  in  performance  and  cost,  but  not  so  much  for  schedule.  In 
previous  AoAs,  we  have  not  done  schedule  trades — just  tried  to  meet  the  given 
schedule.  Trades  are  done  on  technologies,  and  the  effect  on  cost  and 
performance/effectiveness  is  estimated. 

•  Schedule  also  can  be  used  a  discriminator  up  front  to  eliminate  alternatives  that 
cannot  provide  capability  in  the  desired  timeframe. 

•  Schedule  is  the  shortcoming.  The  percentage  change  in  schedule  is  much  higher  than 
the  percentage  change  in  cost.  The  relationship  between  schedule  and  cost  must  be 
understood.  The  predicted  change  in  schedule  is  much  higher  than  the  predicted 
change  in  cost  with  the  element  of  schedule  removed  from  cost. 

•  There  is  a  lack  of  realistic  schedule  estimation.  Realistic  schedules  will  reduce  the 
likelihood  of  cost  and  schedule  breaches. 

The  tradeoff  analysis  should  be  done  before  an  AoA,  upfront  in  the  AoA  process  or 
throughout  the  AoA  process. 

•  Tradeoffs  between  cost,  schedule,  and  technology  should  be  completed  before  an 
AoA  begins. 

•  The  tradeoff  analysis  should  be  done  upfront. 

•  Trades  should  be  done  throughout  the  AoA. 
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•  Tradeoff  analysis  should  not  be  done  6  to  9  months  after  an  AoA. 

The  tradeoff  analysis  will  become  difficult  with  multiple  alternatives  to  consider. 

•  More  fonnal  excursions  can  help  eliminate  trade  candidates. 

•  The  challenge  is  that,  given  too  many  alternatives,  the  tradeoff  analysis  will  become 
burdensome.  For  one  recent  MS  B  AoA,  there  were  12  alternatives.  Performing  cost, 
schedule,  and  technical  risk  assessments  on  all  these  alternatives  is  time-intensive. 
There  are  too  many  alternatives.  The  Army  should  whittle  down  the  alternatives  to 
better  focus  on  what  is  important  for  the  decision-makers. 

Risk  assessments  are  part  of  the  tradeoff  analyses.  The  relationship  between  cost, 
schedule,  and  perfonnance  must  be  understood. 

•  AoAs  are  now  considering  cost,  schedule,  and  technical  risk  analyses.  AMSAA  is 
perfonning  these  risk  assessments. 

•  Risk  assessment  is  part  of  this  tradeoff.  The  relationship  between  cost,  schedule,  and 
perfonnance  must  be  understood.  The  sensitivity  of  changes  in  any  of  these  factors  to 
the  others  must  be  examined.  The  tradeoff  analysis  is  an  iterative  process. 

•  Sensitivity  analysis  should  be  performed  on  the  technology,  cost,  and  schedule  to 
detennine  the  effect  on  performance. 

Risk  methodology  should  have  different  approaches  to  consider  past  systems/components 
and  new  system/components. 

•  The  risk  methodology  should  have  different  approaches  to  consider  past 
systems/components  and  new  system/components.  The  methodology  should  look  into 
the  past  and  to  the  future.  It  is  not  the  same  approach  for  the  past  and  future  estimates 
of  risk.  For  the  past  approach,  the  key  is  the  method  to  establish  analogous  systems. 
Infonnation  about  what  is  known  about  similar  types  of  systems  must  be  compiled. 
For  a  new  system  with  no  analogous  system,  it  should  be  decomposed  into 
parts/subsystems.  Then  some  subsystems  may  have  an  analogous  system,  and  the  past 
method  can  be  used  for  risk.  For  unique  parts,  a  different  approach  must  be  used. 
Integration  should  be  considered. 

•  The  DoD  community  is  still  in  its  infancy  regarding  tradeoff  and  risk  analyses. 

Tradeoff  analyses  allow  more  robust,  cost-effective  solutions  to  be  developed. 


82 


•  The  tradeoff  analysis  allows  the  AoA  team  to  examine  a  more  robust  set  of  ways  to 
solve  the  problem.  In  the  past,  the  AoAs  typically  looked  at  point  solutions. 

•  The  tradeoff  analysis  is  very  important  and  underpins  decisions. 

•  Unlike  AoA  developed  pre-WSARA,  current  AoAs  factor  in  cost  upfront.  I  think  that 
is  one  of  the  key  benefits  of  the  current  AoA  guidance.  Cost  and  affordability 
boundaries  are  established  early  in  the  analysis.  In  the  past,  the  operational  analysis 
was  conducted  parallel  but  separate  to  the  cost  analysis.  The  two  analyses  were 
combined  in  the  final  briefings  and  report.  If  the  system  wasn’t  affordable  in  the 
desired  timeframe,  development  or  procurement  was  stretched.  Similarly,  alternatives 
weren’t  always  vetted  for  schedule  realism. 

The  tradeoff  analyses  affect  the  requirements  generation  process.  Tradeoffs  can  be  used 
to  determine  the  minimum  requirements  to  reduce  cost  and  maintain  schedule.  The  JCIDS 
process  also  should  consider  trades  before  an  AoA  is  conducted. 

•  A  requirement/technology  can  be  traded  to  reduce  cost  or  schedule.  The  tradeoff 
analysis  flows  back  to  the  requirements.  The  requirements  should  not  be  locked  in 
early,  but  they  should  be  influenced  by  the  tradeoff  analysis. 

•  The  AoA  can  detennine  analytically  what  is  the  minimum  technology  and 
perfonnance  to  meet  CDD  requirements  and  decrease  cost  and  maintain  schedule. 

•  AoAs  inform  draft  CDDs,  and  the  tradeoff  analysis  could  help  make  acceptable 
programmatic  and  requirements  tradeoffs.  Cost  is  a  focus  on  the  trades. 

•  The  JCIDS  process  needs  to  consider  the  trades  work  upfront  before  an  AoA  is 
conducted. 

Currently,  cost  is  the  most  important  factor  in  the  decision  space. 

•  Numerous  options  are  investigated  in  AoAs  related  to  extending  schedule, 
increasing/decreasing  capability,  reducing  cost,  etc.  Currently,  cost  is  the  most 
important  factor.  Schedule  also  is  important  to  get  the  equipment  to  the  soldier 
quicker. 

•  For  example,  in  2005,  the  main  focus  was  fielding  systems  quickly  for  the  war. 
Schedule  issues  dominated  the  decision  space.  Currently,  the  cost  of  the  systems  is 
the  most  important  factor. 

For  the  tradeoff  analysis,  LCC  should  be  considered  rather  than  only  procurement  costs. 
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•  For  cost  trades,  LCC  should  be  considered  rather  than  just  the  procurement  costs.  The 
big  driver  should  be  LCC.  Past  AoAs  have  examined  procurement  or  unit  cost. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

The  Anny  should  involve  OSD  CAPE  early  in  the  AoA  process.  The  Anny  must 
understand  OSD  CAPE’s  guidance,  and  OSD  CAPE  must  understand  the  Army’s  proposed  study 
plan.  The  Anny  should  be  proactive  in  writing  draft  AoA  guidance  for  OSD  CAPE’s 
consideration. 

•  OSD  CAPE  should  be  brought  in  early  to  the  Army’s  AoA  process. 

•  The  Anny  should  get  OSD  CAPE  involved  early  and  often. 

•  For  future  AoAs,  the  guidance  should  be  understood  earlier.  Sometimes  OSD  CAPE 
guidance  is  provided  one  day  and  three  days  later  the  AoA  study  plan  is  required. 
Sometimes  the  OSD  CAPE  guidance  is  not  officially  approved  until  months  later. 
There  needs  to  be  quicker  agreement  on  the  study  plan.  There  should  be  earlier 
interaction  with  OSD  CAPE. 

•  There  must  be  interactions  with  OSD  CAPE  early  and  often. 

•  The  Army  needs  to  ask  OSD  CAPE  more  questions  on  the  guidance  and  have  a  better 
understanding  of  the  guidance.  The  Anny  should  engage  OSD  CAPE  to  obtain  an 
agreement  or  expectation  on  what  is  required  in  the  AoA.  If  this  is  done,  OSD  CAPE 
may  solidify  its  guidance  so  it  does  not  change  in  the  1 1th  hour. 

•  The  Army  should  be  proactive  in  writing  draft  AoA  guidance  and  send  this  guidance 
to  OSD  CAPE.  The  Army  needs  OSD  CAPE  guidance  early  so  it  would  not  be 
“marching  south  instead  of  going  north”  as  OSD  CAPE  would  like. 

The  Army  must  increase  the  open  and  transparent  communication  with  OSD  CAPE 
during  the  AoA.  The  Army  must  share  infonnation  and  explain  the  analyses  to  OSD  CAPE  often 
during  the  AoA  process.  The  Army  must  work  closely  with  OSD  CAPE  to  increase  its  level  of 
understanding  of  the  analysis  so  they  understand  that  the  analysis  is  credible  and  relevant. 

•  There  should  be  better  communication  with  OSD  CAPE,  and  it  should  be  open  and 
transparent. 

•  There  should  be  better  communication  with  OSD  CAPE  to  determine  what  they  are 
looking  for. 

•  Increase  the  dialogue  with  OSD  CAPE  during  the  AoA. 
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•  The  guidance  from  OSD  CAPE  is  important.  The  Anny  has  to  share  information  with 
OSD  CAPE  along  the  way.  The  Anny  can’t  wait  until  just  before  an  MS  decision,  but 
it  must  be  willing  to  share  early  on. 

•  OSD  should  be  included  in  all  IPRs  and  meetings  regarding  the  AoAs.  AoAs  should 
be  open  and  transparent. 

•  The  analysis  must  be  transparent  and  open  to  OSD  CAPE. 

•  For  future  AoAs,  the  Army  must  be  transparent  with  OSD  CAPE,  engaging  it  on  the 
details  of  the  analysis — key  limitations  and  assumptions  of  the  analysis. 

•  The  Anny  must  work  closely  with  OSD  CAPE  to  increase  its  level  of  understanding 
of  the  analysis  so  the  analysis  is  credible  and  relevant.  OSD  CAPE  has  to  understand 
what  is  “under  the  hood.” 

•  The  Anny  must  communicate  with  and  educate  OSD  CAPE. 

•  The  Anny  has  to  work  closer  with  OSD  to  detennine  what  is  acceptable. 

•  The  Anny  must  have  the  ability  to  work  well  with  OSD  CAPE.  The  analysis  process 
must  be  transparent  and  inclusive.  Trust  must  be  established  in  both  directions.  The 
Anny  has  to  go  into  the  process  with  the  expectation  that  OSD  CAPE  is  in  the  right 
place  and  wants  to  do  the  right  thing.  OSD  CAPE  must  have  the  same  expectation  of 
the  Army. 

The  Anny  must  ensure  the  AoA  results  are  presented  in  an  unbiased  manner.  The  Army 
should  involve  OSD  CAPE  in  the  interpretation  of  results.  There  should  be  no  preferred 
alternatives  in  AoAs. 

•  OSD  CAPE  believes  the  AoA  team  may  not  have  the  freedom  to  explain  the  results 
of  its  analyses.  OSD  CAPE  thinks  there  is  a  kind  of  coercive  effect  on  what  the 
analyses  say.  OSD  CAPE  believes  analysis  may  be  good  (i.e.,  quality  and 
objectivity),  but  interpret  the  data  different  than  how  the  Anny  does.  The  Army  tends 
to  compare  a  new  system  vs.  the  competitors,  while  OSD  CAPE  looks  at  a  different 
decision  choice.  The  Army  has  to  involve  OSD  CAPE  in  the  interpretation  of  results. 

•  For  Pre-MS  A,  there  has  been  an  issue  of  an  Army-preferred  alternative  in  past  AoAs. 
There  usually  is  significant  work  associated  with  packaging  this  preferred  alternative, 
and  there  would  be  less  effort  in  defining  the  other  alternatives.  Resolution  of  detail 
would  not  be  equal  in  defining  these  alternatives.  It  was  more  difficult  to  develop  the 
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cost  estimates,  and  to  have  the  same  level  of  quality  in  the  cost  estimates,  for  the 
nonproferred  alternatives. 

•  In  the  past,  all  the  analysis  results  tended  to  support  the  preferred  alternative.  Not 
having  a  preferred  alternative  would  improve  the  quality  of  the  analysis  to  support 
decision-makers. 

AoA  analyses  and  results  must  be  related  to  OSD  CAPE  decision  choices.  The  decision 
choices  are  not  necessarily  alternatives.  AoAs  must  inform  and  enable  decisions  and  be  focused 
on  the  decision  space. 

•  AoAs  failed  to  speak  to  OSD  CAPE  in  its  language.  The  Army  has  to  learn  to  speak 
to  OSD  CAPE  in  ways  relevant  to  its  decision  choices.  Decision  choices  are  not 
necessarily  alternatives.  The  Anny  has  to  cast  results  in  terms  of  OSD  CAPE  decision 
choices. 

•  Anny  AoAs  should  be  framed  in  the  context  of  the  decisions.  The  analysis 
community  does  not  care  about  the  outcome,  and  it  is  not  a  proponent  for  any 
alternative.  There  is  no  predetennined  alternative. 

•  AoAs  must  infonn  and  enable  decisions.  AoAs  must  address  how  to  fill  the  gap  with 
what  is  available  to  the  Army.  For  the  Anny,  the  soldier  is  the  ultimate  customer. 

•  The  AoAs  must  better  address  the  decision  space. 

•  The  AoA  should  focus  on  the  decision-makers’  expectations.  The  AoA  team  should 
brief  the  decision-makers  often  to  obtain  their  expectations  and  guidance  and  to 
ensure  that  the  AoA  is  on  the  right  path. 

•  AoAs  should  provide  the  best  set  of  alternatives  for  the  decision-maker. 

There  should  be  an  agreed-upon  set  of  alternatives  and  an  established  hierarchy  within 
this  set  to  ensure  analyses  are  perfonned  on  them. 

•  There  should  be  a  hierarchy  to  the  alternatives  to  ensure  analysis  is  performed  on  the 
important  alternatives. 

•  There  needs  to  be  an  agreed-upon  set  of  alternatives  for  full  treatment. 

•  There  is  a  lack  of  high-level  guidance  for  AoAs. 

•  In  future  AoAs,  well-defined  alternatives  would  be  beneficial.  Sometimes  alternatives 
are  not  defined  to  a  degree  to  estimate  LCCs. 
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The  scope  of  an  AoA  should  be  reduced  as  it  progresses.  The  number  of  alternatives 
should  be  reduced  by  the  SAG  where  alternatives  are  no  longer  feasible  or  do  not  add  to  the 
decision  choices.  Analyses  that  do  not  add  information  relevant  to  the  decision  should  be 
stopped. 

•  Angst  and  frustration  arise  on  the  scoping  of  the  AoAs  (scope  can  increase  and  get 
out  of  hand). 

•  AoA  must  be  completed  quicker.  The  AoA  timeline  has  to  fit  the  schedule.  The 
strategy  should  be  to  reduce  the  scope  of  the  study  as  it  progresses.  The  AoA  may 
start  out  with  a  number  of  alternatives  and  analysis  planned,  but  at  the 
IPRs/knowledge  points,  some  things  may  be  dropped  off  table.  This  reduction  of 
scope  would  help  the  team  complete  the  study.  In  the  past,  things  (alternatives  or 
analyses)  always  get  added  but  nothing  gets  taken  off. 

•  There  is  a  point  of  diminishing  returns  on  alternatives. 

•  AoAs  should  have  a  reasonable  number  of  alternatives,  and  alternatives  should  not  be 
added,  but  rather  excursions  made. 

•  Early  excursion  alternatives — there  is  a  need  to  know  when  to  stop  analysis  on  them. 

•  Know  when  enough  is  enough  as  opposed  to  follow-on  analysis. 

•  AoAs  can  form  a  life  of  its  own  (analyses  breed  analyses). 

AoAs  must  explore  trade  space  for  cost,  schedule,  and  performance.  Risk  assessment 
should  be  conducted  for  cost,  schedule,  and  technical  (including  integration). 

•  AoAs  must  fully  explore  the  trades  space.  The  items  that  are  open  for  trades  should 
be  identified. 

•  Cost-informed  trades  should  be  part  of  the  AoAs. 

•  The  AoA  team  has  to  educate  the  user  community  and  COEs  on  the  tradeoff  analyses. 

•  Tradeoff  and  risk  analysis  need  to  be  performed.  Technical  risk  can  be  used  as  a 
screening  criterion.  Caution  is  needed,  because  some  technology  advocates  may  claim 
the  TRL  is  higher  than  it  actually  is. 

•  Risk  assessment  should  be  conducted  for  cost,  schedule,  and  technical  (including 
integration).  Risk  analyses  are  done  at  MS  A  and  MS  B,  but  in  more  detail  for  MS  A. 
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•  Cost  is  a  big  issue.  For  future  AoAs,  the  team  should  work  with  the  users  to  reduce 
cost  through  the  tradeoff  analyses.  For  some  systems,  the  question  is:  What  can  we 
live  with? 

•  The  cost,  schedule,  and  perfonnance  trades/risk  analysis  should  be  part  of  the  main 
analysis  and  not  performed  at  the  end. 

•  There  is  a  lack  of  TRL  data  for  AoAs.  The  AoA  teams  must  rely  on  SMEs.  Much  of 
the  cost  growth  in  systems  is  due  to  the  immaturity  of  alternatives,  and  the  lack  of 
good  TRL  estimates. 

•  There  also  is  a  risk  of  the  performance  not  translating  into  operational  reality. 

A  coherent,  robust  risk  and  trade  methodology  must  be  developed  by  the  Anny.  AMSAA 
is  currently  working  on  this  approach,  and  DASA-CE  has  a  tool  to  support  trades. 

•  AMSAA  must  develop  a  robust  risk  assessment  approach. 

•  A  coherent  approach  and  methodology  are  needed  for  risk  assessment.  Some 
organization  needs  to  be  the  one  that  says  what  the  risk  is.  Manufacturing  risk  is 
difficult  for  foreign  systems. 

•  Additional  methodology  work  is  needed  in  the  cost,  schedule,  and  performance  trades 
and  risk  analyses. 

•  The  requirements,  schedule,  cost,  and  perfonnance  need  to  integrate  to  perfonn  the 
trades  and  risk  analyses.  The  DASA-CE  Joint  Integrated  Analysis  Tool  (JIAT)  shows 
the  results  of  excursions  or  trades.  This  tool  would  help  expedite  the  trades  analysis, 
and  create  greater  cohesion  with  the  AoA  work  group.  DASA-CE  is  well  situated  to 
help  with  this  analysis.  The  analysis  community  should  take  a  closer  look  at  the  JIAT. 
More  work  needs  to  be  done  with  the  models  and  databases  within  JIAT. 

•  Cost  goals  are  needed  to  help  with  the  trades. 

Cost  methodology  should  be  refined  to  develop  more  estimating  relationships  that  link 
perfonnance  to  cost,  better  incorporate  risk,  and  consider  affordability. 

•  From  the  cost  analysis  perspective,  we  will  have  to  develop  more  estimating 
relationships  that  can  link  the  capability/performance  to  cost  to  facilitate  quicker  and 
more  accurate  cost  trades. 

•  We  also  have  to  better  incorporate  risk  into  our  cost  estimates. 
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•  Affordability  should  be  considered,  especially  the  affordability  of  the  life  of  the 
system.  Current  acquisition  considers  the  affordability  over  the  POM  years,  but  does 
not  consider  O&S  or  disposal  costs.  LCCs  should  be  considered. 

Conducting  fonnal  AoAs  for  each  MS  decision  would  increase  the  fidelity  and  structure 
of  the  analysis  and  would  increase  the  information  flow  for  the  analyses. 

•  More  formal  AoA  should  be  conducted  instead  of  less  formal  analysis.  Without 
AoAs,  analysis  may  not  be  done  at  each  MS,  there  is  less  rigorous  ongoing  analysis, 
and  analysis  is  not  reported  to  fonnal  SAGs.  Having  a  formal  AoA  at  each  MS  would 
allow  for  a  constant  AoA  process  and  better  flow  of  information. 

The  MS  A  AoA  results  must  be  used  to  develop  the  CDD. 

•  The  CDD  should  not  be  developed  before  the  MS  A  AoA  is  completed. 

•  The  JROC  is  interested  in  the  AoA  process.  It  needs  the  AoA  to  support  CDD 
decisions.  The  AoA  would  show  importance  in  operational  capability. 

There  are  funding  challenges  associated  with  the  conduct  of  AoAs. 

•  There  needs  to  be  funding  to  conduct  AoAs.  The  PM/PEOs  have  paid  in  the  past,  but 
there  should  be  some  mission  funding  associated  with  AoAs. 

•  There  are  funding  challenges  for  the  conduct  of  AoAs. 

There  should  be  realistic  schedules  for  conducting  AoAs. 

•  There  should  be  a  realistic  schedule  associated  with  conducting  an  AoA.  In  the  past, 
the  AoA  schedule  has  been  accelerated  too  much. 

The  Anny  needs  better  unity  of  purpose. 

•  The  Army  needs  to  decide  what  we  need  to  make  decisions.  Better  unity  of  purpose 
within  the  Anny  is  needed.  Internal  Army  dissention  should  not  be  raised  at  the  SAG, 
but  rather  it  should  have  been  hashed  out  at  the  Army  level  beforehand. 

•  The  loss  of  the  DUSA-OR  [Deputy  Under  Secretary  of  the  Army  (Operations 
Research)]  has  hurt  the  AoA  process.  Before,  the  DUSA-OR  interfaced  with  the  OSD 
secretariat  at  the  three-star  level.  Now  the  Anny  Staff  including  the  G-8  and  ARCIC 
three  stars  have  to  talk  to  the  DOSD  CAPE.  The  DUSA-OR  had  the  continuity  and 
the  position  on  the  Army  secretariat.  It  is  better  to  deal  with  OSD  CAPE  at  the  Army 
Secretary  level  instead  of  the  Staff  level. 
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The  Army  should  examine  Navy  and  Air  Force  AoAs  that  OSD  CAPE  believes  are  well 

done. 

•  Examine  AoAs  that  OSD  CAPE  believes  were  done  well. 

•  Learn  how  the  Navy  and  Air  Force  are  doing  AoA,  and  what  they  are  doing 
differently  than  the  Anny.  We  should  learn  from  them. 

Other  recommendations  were  made  to  improve  AoAs. 

•  The  soldier  is  the  ultimate  customer.  The  objective  is  to  get  the  best  possible 
equipment  into  the  hands  of  the  soldier.  All  the  stakeholders  share  this  same  ultimate 
goal  of  doing  what  is  best  for  the  soldiers,  sailors,  and  marines. 

•  The  analysis  community  does  not  have  a  “dog  in  the  fight”  and  can  recommend  the 
best  investment  decision. 

•  Portfolio  and  fleet  size  analysis  may  be  required  in  AoAs. 

•  Requirements  should  be  defined  so  the  AoA  can  determine  if  the  performance  meets 
the  requirements. 

•  Help  is  needed  on  what  to  trade. 

•  AoAs  should  be  done  earlier  (once  the  JCIDS  process  determines  a  materiel  solution 
is  required)  and  repeated  later  on  if  needed.  The  later  AoAs  would  have  more  details. 
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APPENDIX  B 

ARMY  AOA  OVERSIGHT  INTERVIEW  RESULTS 

This  section  provides  the  answers  for  personnel  from  the  Anny  organizations  DAMO-CI, 
DCS  G-3/5/7  and  ARCIC  (D.  Markowitz,  personal  communication,  March  2,  2012;  A.  Resnick, 
personal  communication,  March  12,  2012;  R.  Hume,  personal  communication,  March  16,  2012; 
R.  Holdren,  personal  communication,  February  16,  2012;  and  M.  Moore,  personal 
communication,  March  2,  2012).  These  organizations  provide  oversight  to  Anny  AoAs.  The 
answers  are  compiled  for  each  question.  For  each  question,  the  interviewees’  answers  are 
grouped  by  general  ideas.  A  summary  of  the  key  points  for  all  interviewees  are  provided 
followed  by  the  interviewees’  specific  comments  as  bullet  points. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

AoAs  should  support  the  requirements  development  process.  AoAs  should  also  support 
multiple  customers  in  the  requirements  development  and  acquisition  communities. 

•  AoAs  help  the  requirements  process,  and  the  results  are  codified  in  the  requirements. 
From  MDD  to  MS  A,  the  requirements  are  being  developed.  Past  MS  A,  the 
specification  is  being  developed  and  the  program  baseline  is  being  developed  in  terms 
of  cost  and  schedule.  The  AoA  results  feed  multiple  customers.  AoAs  assist  in  the 
approval  process  at  MS  decisions,  and  they  are  supposed  to  feed  the  ongoing  process 
to  develop  requirements  and  specifications. 

•  Recent  programs  have  knowledge  points  in  which  the  AoA  can  help  the  acquisition 
process.  The  ICDT  is  a  new  team  that  follows  the  requirements  through  the 
knowledge  point  process. 

•  The  use  of  the  tenn  “alternative”  often  confuses  what  constitutes  the  main  purpose  of 
an  AoA.  AoAs  are  not  meant  to  be  source  selections,  rather  they  support  the 
requirements  process  and  specifically  the  development  of  the  CDD. 

OSD  CAPE  is  focused  on  supporting  MS  decisions  with  AoAs  rather  than  supporting 
requirements  development  or  acquisition  processes. 

•  OSD  CAPE  is  focused  on  supporting  the  MS  decision  process  with  AoAs.  The 
organizations  conducting  the  AoAs  are  focused  on  checking  the  box  for  the  MS 
process  rather  than  helping  the  requirements  or  acquisition  processes. 


91 


AoAs  should  inform  decisions  regarding  acquisition  (including  cost  and  schedule), 
requirements,  and  resource  allocation. 

•  AoAs  are  COEAs  conducted  for  the  purpose  of  infonning  decisions  regarding 
acquisition,  requirements,  and  resource  allocation/prioritization. 

•  The  purpose  of  an  AoA  is  to  inform  the  decision-makers  for  an  acquisition  decision. 

•  AoAs  also  affect  decisions  on  cost  and  schedule. 

Formal  AoAs  at  the  later  MS  decisions  are  needed  to  support  all  the  decision  points 
throughout  the  acquisition  process.  There  are  too  many  unknowns  early  in  the  program  for  an 
MS  A  AoA  to  answer  all  the  questions. 

•  Before  WSARA,  AoAs  supported  the  MS  B  decision  point.  Now,  OSD  wants  one 
AoA  that  covers  all  decision  points  (MS  A  to  MS  C).  OSD  wants  to  avoid  carrying  an 
expensive  program  to  the  MS  C  decision  only  to  cancel  it.  The  system  may  be 
canceled  because  it  either  is  too  expensive  or  can’t  meet  the  specifications.  OSD 
wants  to  learn  as  much  as  possible  early  in  the  acquisition  process.  One  AoA 
covering  all  decision  points  cannot  achieve  that  vision.  There  are  too  many  unknowns 
at  the  beginning  of  an  AoA,  especially  if  an  AoA  starts  at  the  MS  A  decision  point. 

The  purpose  of  an  AoA  is  to  develop  the  supporting  evidence  to  identify  an  achievable, 
affordable,  and  operationally  relevant  set  of  attributes.  Alternatives  are  the  “stalking  horses”  to 
better  understand  the  set  of  attributes.  Alternatives  are  the  surrogate  system  that  allows  analysis 
to  be  conducted  on  a  set  of  attributes. 

•  The  purpose  of  an  AoA  is  to  develop  the  supporting  evidence  to  identify  an 
achievable,  affordable,  and  operationally  relevant  set  of  attributes.  The  name  AoA  is 
a  misnomer,  and  the  use  of  the  tenn  “alternative”  is  problematic.  Using  the  term 
alternative  implies  that  solutions  have  already  been  identified.  Alternatives  are  the 
“stalking  horses”  to  better  understand  the  set  of  attributes.  The  alternatives  would  be 
used  to  test  the  concept  of  a  particular  set  of  attributes.  The  analysis  is  not  to 
detennine  the  best  alternative,  but  to  identify  an  achievable,  affordable,  and 
operationally  relevant  set  of  attributes. 

•  While  designated  alternative  systems  are  used  (either  an  existing  system  or  a 
conceptual  design),  the  alternative  is  used  as  a  host  or  surrogate  system  that  allows 
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analysis  to  be  focused  on  those  attributes  a  given  system  may  have.  Ensuring  that  the 
operational  requirement  is  fully  understood,  the  AoA  conducts  parametric  trades 
analysis  to  identify  the  set  of  key  attributes  that  are  operationally  acceptable 
(minimum  necessary  capability  to  accomplish  specific  mission),  feasible  (technology 
is  possible  in  specified  timeframe),  and  affordable  (cost  within  established 
affordability  target)  that  should  be  identified  in  the  updated  capability  document 
(usually  the  CDD). 

2.  What  do  you  think  are  the  major  issues  associated  with  AoAs  under  the  new  WSARA 
guidance? 

Completing  an  AoA  at  MS  A  is  too  early,  but  completing  a  detailed  AoA  at  MS  B  is  too 
late.  The  correct  timing  for  an  AoA  is  at  the  PDR. 

•  There  have  been  multiple  reforms  to  the  acquisition  process.  These  reforms  indicate 
there  is  not  a  good  understanding  of  the  timing  and  what  infonnation  is  needed  from 
AoAs.  WSARA  requires  additional  analyses  from  the  MS  A  AoA.  There  is 
uncertainty  on  the  difference  between  what  is  required  from  AoAs  for  MS  A  vs.  MS 
B.  Completing  a  detailed  AoA  at  MS  A  is  too  early,  but  completing  a  detailed  AoA  at 
MS  B  is  too  late.  The  correct  timing  for  an  AoA  is  at  the  PDR.  The  cost-informed 
trades  perfonned  in  the  AoA  need  to  feed  the  design  reviews.  The  AoAs  should  not 
be  grading  the  PDR  results  at  MS  decision  points. 

AoAs  should  not  try  to  answer  questions  already  being  answered  in  the  acquisition 
process  (e.g.,  source  selection). 

•  There  is  a  relationship  between  AoAs  and  the  acquisition  strategy.  AoAs  should  not 
be  used  for  source  selection,  and  they  should  not  try  to  answer  questions  that  the 
acquisition  process  will  answer. 

The  requirements  and  acquisition  communities  also  are  customers  of  AoAs,  in  addition  to 
OSD  CAPE.  AoA  should  benefit  all  stakeholders  and  not  only  focus  on  OSD  CAPE. 

•  Again,  the  question  is:  Who  is  the  customer?  There  are  many  customers  of  AoAs. 
OSD  is  a  customer  for  the  MS  decisions,  and  the  requirements  community  is  a 
customer  for  the  requirements  development  process.  The  acquisition  community  is  a 
customer,  as  well  as  organizations  that  are  developing  fleet-mix  strategies. 
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•  The  tendency  of  WSARA  is  to  focus  AoAs  on  OSD  CAPE  and  not  assist  the  materiel 
development  process.  AoAs  should  help  the  process  and  not  judge  the  decisions. 

•  Establishing  common  view  among  all  stakeholders  regarding  the  purpose  and  scope 
of  AoAs  is  a  challenge. 

Capability  gaps  and  CONOPS  should  be  developed  before  the  start  of  an  AoA. 

•  Gaps  and  CONOPS  should  be  developed  before  materiel  solution  is  pursued. 
TRADOC  performs  gap  analysis  and  DOTMLPF  analysis  to  determine  if  a  materiel 
solution  is  required.  The  CONOPS  begins  at  this  time.  If  the  gaps  and  CONOPS  are 
not  well  defined  before  the  materiel  solution  is  pursued,  the  acquisition  process  may 
not  be  effective. 

Timely  OSD  guidance  and  adequate  AoA  study  plans  are  needed. 

•  Development  of  timely  OSD  guidance  is  a  challenge. 

•  Development  of  adequate  AoA  study  plans  prior  to  the  MDD  is  a  challenge. 

•  AoAs  are  not  cookie  cutters,  and  they  must  be  tailored  to  address  the  specific 
problems. 

WSARA  gave  OSD  CAPE  more  influence  on  AoAs,  because  OSD  CAPE  has  to  declare 
an  AoA  sufficient  to  support  MS  decisions. 

•  One  major  change  in  the  WSARA  legislation  is  that  it  now  requires  OSD  CAPE  to 
declare  an  AoA  sufficient  before  a  program  can  pass  a  MS  decision  point.  Before 
WSARA,  an  AoA  fell  under  the  Title  10  responsibility  of  each  of  the  Services  with 
OSD  providing  oversight  and  policy  guidance.  WSARA  shifts  the  responsibility  of 
declaring  an  AoA  sufficient  from  the  Service  to  OSD. 

•  WSARA  gives  guidance  to  OSD  CAPE  on  what  it  should  evaluate  before  declaring 
sufficiency. 

•  WSARA  of  2009  empowered  OSD  CAPE  in  the  realm  of  AoAs. 

WSARA  specifies  that  the  AoA  should  cover  cost,  schedule,  and  performance  tradeoffs. 
Requirements  can  be  traded. 

•  WSARA  specifies  that  the  AoA  should  cover  the  trade  space  among  cost,  schedule, 
and  perfonnance. 

•  There  is  more  focus  on  cost,  schedule,  and  perfonnance  trades.  These  are  trades  to 
requirements,  and  the  focus  is  now  on  the  requirement.  There  is  also  risk  associated 
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with  making  the  trades.  For  older  AoAs,  the  requirements  were  accepted  as  firm.  For 
AoAs  under  WSARA,  the  requirements  must  be  justified,  defended,  and  proven  valid. 
Requirement  can  be  traded. 

Army  AoAs  have  increased  emphasis  focus  on  measurement  space  and  cost-benefit 
analyses. 

•  TRAC  (main  proponent  for  major  program  AoAs)  has  increased  emphasis  on 
conducting  appropriate  measurement  space  analysis  at  the  beginning  of  every  study. 
The  measurement  space  analysis  brings  together  the  appropriate  analysts,  and  user 
representatives  address  the  following:  clearly  define  the  problem  (what  is  the 
capability  gap  that  must  be  addressed?);  understand  the  capability  (what  capability  is 
needed  to  address  gaps  given  detailed  review  of  operational  task,  condition,  and 
standards);  determine  what  constitutes  the  critical  attributes;  and  determine  trades  that 
can  be  made  to  meet  affordability  targets  (what  can  be  traded  to  reduce  cost  but 
enhance  capability).  This  leads  to  more  focused  trades  analysis  so  the  Army  gets 
more  “bang  for  the  buck.”  In  general,  more  emphasis  is  being  placed  on  cost-benefit 
analysis. 

WSARA  has  increased  the  workloads  of  the  analytical  community,  and  the  Army  may 
not  have  the  resources  (funding  or  personnel)  to  perform  detailed  AoAs.  A  central  funding 
source  should  be  available  for  AoAs. 

•  WSARA  has  increased  the  workload  on  the  analytical  community. 

•  The  Army  does  not  have  the  resources  (funding  or  personnel)  to  perform  the  level  of 
detail  in  AoAs.  There  should  be  a  central  funding  source  for  the  organizations 
performing  AoAs. 

The  Army  does  not  have  enough  analysts  s  killed  in  technical  risk  assessment,  cost 
analyses,  or  affordability  assessments. 

•  A  primary  focus  of  major  program  AoAs  is  on  technical  risk  and  costs.  However,  the 
Army  does  not  have  enough  analysts  with  the  right  skills  and  expertise  to  address  cost 
and  affordability  to  a  sufficient  level  of  detail  required. 

•  The  Army  must  rely  on  a  few  organizations  to  perform  the  AoAs  in  order  to  train  and 
develop  the  skilled  team  (e.g.,  AMSAA  and  TRAC). 
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3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

AoA  must  be  tailored  to  the  specific  program,  requirements,  and  problems. 

•  It  would  depend  on  the  program  and  the  requirements.  Before  MDD,  the  acquisition 
community  does  not  know  what  MS  the  program  will  enter  the  acquisition  process. 

•  AoAs  are  not  cookie  cutters,  and  they  must  be  tailored  to  address  the  specific 
problems. 

The  timing  of  AoAs  is  not  well-coordinated  with  the  acquisition  process.  Cost,  schedule, 
and  performance  trades  at  MS  A  are  too  early  and  at  MS  B  are  too  late;  the  PDR  would  be  better 
timing  for  an  AoA. 

•  Anny  AoAs  are  not  well-correlated  with  the  acquisition  process.  A  detailed  AoA  at 
MS  A  is  too  early  and  too  late  at  MS  B.  The  PDR  would  be  a  better  timing  of  an 
AoA.  In  theory,  a  PDR  should  be  conducted  6  months  before  MS  B.  AoAs  are  used  to 
support  the  yes-no  MS  decision  process.  The  AoAs  also  are  supposed  to  help  the  cost, 
schedule,  and  performance  trades,  but  this  is  required  earlier  than  MS  B.  For  a  pre- 
MS  A  AoA,  the  consideration  of  cost,  schedule,  and  perfonnance  is  too  early. 

It  is  not  practical  to  have  one  AoA  support  all  MS  decisions. 

•  Based  on  OSD,  there  is  one  AoA  to  support  all  MS  decisions,  which  is  not  practical. 
At  the  earlier  MS  decisions,  there  are  insufficient  knowledge,  data,  and  prototypes. 
The  perfonnance  data  are  assumed,  because  the  system  is  not  developed  (no 
prototype)  and  not  tested. 

There  should  not  be  an  AoA  associated  with  every  MS. 

•  There  should  not  be  an  AoA  associated  with  every  MS. 

•  AoA  should  only  be  performed  after  MS  A  by  exception.  The  Army  does  not  have  the 
resources  to  perform  AoAs  for  every  MS. 

•  AoAs  are  supposed  to  be  performed  for  MS  A  and  then  potentially  at  MS  B  or  MS  C, 
if  needed.  The  reasons  for  requiring  an  AoA  update  would  be  if  more  information  is 
required  (i.e.,  a  deeper  dive  is  required)  or  more  trades  need  to  be  performed. 

Before  MDD,  pre-AoA  analyses  must  identify  the  capability  gaps,  threats,  problems, 
attribute  trade  space,  and  feasible  alternatives. 

•  For  the  MDD,  the  AoA,  or  pre-AoA  analysis,  must  identify  the  capability  gaps  and 
threats  that  will  fonn  the  foundation  of  the  AoA  effort. 
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•  For  an  MDD,  for  example,  the  “Pre-AoA”  effort  must  be  focused  on  defining  the 
problem,  understanding  the  requirement,  identifying  attribute  trade  space,  and 
identifying  the  feasible  alternatives.  If  an  AoA  plan  has  to  be  approved  prior  to  the 
MDD  (as  is  the  case  for  major  AC  AT  I  programs),  the  “Pre-AoA”  must  be  started  as 
soon  as  a  valid  requirements  document  is  approved.  More  emphasis  is  needed  to 
define  the  problem  and  understand  the  requirement. 

For  MS  A,  the  AoA  should  inform  the  requirements  by  evaluating  the  cost,  schedule, 
perfonnance,  and  risk  of  the  concepts;  determining  a  set  of  achievable,  affordable,  and 
operationally  relevant  attributes  for  the  system;  and  identifying  affordable,  operationally 
acceptable  solutions. 

•  For  MS  A,  the  AoA  should  be  tailored  to  evaluate  alternative  concepts  through  the 
lens  of  cost,  performance,  schedule,  and  risk.  The  AoA  should  also  be  tailored  to 
inform  the  identification  of  appropriate  requirements  thresholds  and  objectives  for 
KPPs  and  KSAs. 

•  For  MS  A,  the  central  issue  is  detennining  a  set  of  achievable,  affordable,  and 
operationally  relevant  attributes  for  the  system,  and  these  attributes  must  feed  the 
CDD — ensuring  there  is  a  defensible  basis  for  establishing  these  attributes. 

•  At  MS  A,  the  AoAs  should  assess  alternatives  to  identify  the  best  value  set  of  feasible 
solution  attributes  that  are  operationally  acceptable  and  affordable.  This  is  the  classic 
AoA  that  supports  the  requirements  process  and  is  used  to  refine  the  draft  CDD. 

For  MS  B,  the  AoA  should  identify  cost  drivers,  conduct  cost-performance  trades, 
support  CDD  development,  and  support  the  PDR.  The  AoA  should  also  answer  the  question  of 
whether  or  not  to  build  the  system. 

•  For  MS  B,  the  AoA  should  be  tailored  to  leverage  infonnation  obtained  from  the 
technology  demonstration  activities  in  order  to  identify  cost  drivers;  support  cost 
perfonnance  tradeoffs;  support  development  of  threshold  and  objective  KPP  and  KSA 
values;  and  support  PDR.  For  MS  B,  the  AoA  also  should  be  tailored  in  order  to 
support  the  development  of  the  Basis  of  Issue  (BOI). 

•  If  there  is  a  need  for  an  AoA  at  MS  B  (and  that  is  an  exception),  the  central  question 
is  to  build  or  not  to  build  the  new  system.  An  AoA  at  this  MS  would  answer  any 
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questions  that  were  unanswered  in  the  MS  A  AoA  or  if  the  cost  or  risks  (cost, 
schedule,  or  performance)  changed  significantly. 

At  MS  C,  the  AoA  support  provides  infonnation  to  support  a  fielding  decision  for  the 
system  and  address  affordable  distribution  of  the  system. 

•  For  MS  C,  the  key  question  is  to  field  or  not  to  field;  the  Army  has  the  system  in 
hand.  Again,  if  there  is  driving  need,  the  AoA  could  address  the  most  affordable 
distribution,  but  this  is  more  of  an  economic  analysis. 

AoAs  should  identify  the  cost  of  capabilities  and  how  much  the  capabilities  can  be 
reduced  to  make  the  system  more  affordable. 

•  Cost  is  now  a  major  driver  and  the  following  questions  are  asked:  “Can  we  afford  that 
much  capability?  How  much  can  the  capability  be  reduced  to  lower  the  cost  of  the 
system?”  All  requirements  in  approved  documents  are  open  to  scrutiny  and  potential 
change,  because  the  users  may  revisit  risk,  based  on  projected  costs.  It  is  important  to 
establish  minimum  requirement  thresholds  to  make  sure  the  original  requirement  is 
not  whittled  down  to  the  point  where  it  is  no  longer  worth  pursuing  a  new  materiel 
solution. 

4.  What  types  of  analysis  do  you  think  needs  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 

Gap  analyses  must  be  conducted  to  provide  a  basis  for  the  AoA.  The  AoA  must  assess 
various  ways  to  fill  the  gap. 

•  Sound  gap  analyses  must  be  conducted  in  order  to  provide  a  sound  basis  for  the 
conduct  of  the  AoA. 

•  OSD  CAPE  expectations  are  different  from  classic  AoAs.  OSD  CAPE  is  interested 
primarily  in  validating  the  capability  gaps.  It  is  asking  whether  the  proposed  materiel 
solution  is  really  needed.  It  also  wants  an  exhaustive  analysis  to  enumerate  multiple 
ways  to  achieve  a  necessary  capability.  OSD  CAPE  is  questioning  the  gaps,  and  it 
wants  to  make  sure  the  gaps  are  valid  and  essential.  It  also  wants  the  AoA  to  assess 
what  other  elements  in  the  Joint  Force  can  help  meet  these  gaps. 

Comprehensive  cost,  schedule,  and  performance  tradeoff  analyses  should  be  conducted  in 
an  AoA.  Risk  analyses  should  also  be  conducted.  Risk  estimates  can  be  associated  with  the  given 
cost,  schedule,  and  performance,  or  it  can  be  associated  with  the  trades. 
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•  Comprehensive  cost,  schedule,  and  perfonnance  tradeoff  analyses  are  essential. 

•  A  trade  space  analysis  of  cost,  schedule,  and  performance  should  be  conducted  in  an 
AoA.  These  trades  would  be  made  on  the  requirements. 

•  Risk  analysis  should  also  be  completed.  One  type  of  risk  analysis  is  the  risk 
associated  with  the  given  cost,  schedule,  and  performance.  Another  risk  examines  the 
effect  of  the  cost,  schedule,  and  perfonnance  trades. 

•  The  previous  AoAs  built  the  alternatives  to  requirements  and  then  determined  the 
cost.  Under  WSARA,  the  AoAs  now  trade  requirements  to  detennine  perfonnance  is 
acceptable  to  meet  cost  or  schedule  objectives. 

•  Parametric  and  risk  analyses  should  be  performed  in  AoAs.  Cost  is  a  major  part  of 
these  analyses. 

•  AoAs  should  address  the  risks  associated  with  the  cost,  schedule,  and  performance  of 
the  system.  The  interaction  of  all  three  elements  of  risk  must  be  addressed  as  the  set 
of  attributes  is  finalized. 

AoAs  should  concentrate  on  determining  a  set  of  achievable,  affordable,  and 
operationally  relevant  attributes  that  mitigate  an  operational  problem. 

•  The  AoA  should  address  Anny  needs  and  not  just  OSD  CAPE  needs.  The  Army 
requires  a  set  of  achievable,  affordable,  and  operationally  relevant  attributes  that 
mitigate  an  operational  problem. 

•  The  AoA  should  concentrate  on  the  attributes  (characteristics),  and  affordability  of 
the  system. 

AoAs  should  develop  cost  estimates,  including  APUC,  AMUC  and  LCC. 

•  The  AoA  must  develop  costs,  which  include  APUC  (or  AMUC)  and  LCC.  The  AoA 
should  also  address  system  affordability  to  ensure  the  Army  stays  within  its  means. 

AoA  should  not  address  the  acquisition  strategy,  the  number  of  contractors,  how  to  build 
the  system,  or  the  Technology  Demonstration  feeds  into  engineering  design. 

•  AoAs  should  avoid  being  a  program  management  review — for  example,  not  getting 
into  acquisition  strategy,  the  number  of  contractors,  how  to  build  the  system,  and  the 
Tech  Demo  feeds  into  engineering  design. 
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It  is  more  difficult  for  the  Army  than  the  Air  Force  or  Navy  to  meet  OSD  CAPE 
expectations.  The  Air  Force  and  Navy’s  materiel  solutions  are  more  closely  related  to  the 
strategic  level  than  the  Army’s.  OSD  CAPE  is  more  focused  at  the  strategic  level. 

•  It  is  more  difficult  for  the  Army  than  the  Air  Force  or  Navy  to  meet  OSD  CAPE 
expectations.  For  the  Air  Force  and  Navy,  the  materiel  solution  is  more  closely 
related  to  the  strategic  level.  For  the  Anny,  the  materiel  solution  is  more  closely 
related  to  the  tactical  level,  which  is  several  steps  removed  from  the  strategic  level. 
The  Air  Force  and  Navy  vignettes  are  more  related  to  the  military  and  defense 
strategy,  while  the  Army’s  vignettes  are  more  tactical.  The  relationship  of  Anny 
vignettes  to  the  military  or  defense  strategy  is  unclear.  OSD  CAPE  and  G-3  work 
more  at  the  strategic  level  rather  than  the  tactical  level.  TRAC  does  not  work  at  the 
strategic  level  for  AoAs.  The  TAA  from  Center  for  Anny  Analysis  (CAA)  is  at  the 
strategic  level,  but  this  analysis  is  not  associated  with  AoAs.  Since  OSD  CAPE  is 
more  focused  on  the  strategic  level,  the  Army  AoAs,  which  are  focused  on  the  tactical 
level,  have  more  difficulty  in  meeting  OSD  CAPE’s  expectations. 

•  For  the  Navy,  OSD  CAPE  is  interested  in  the  fleet  needs.  These  needs  are  related 
closely  to  the  National  Defense  Strategy  and  the  National  Military  Strategy.  Cost  and 
schedule  are  also  part  of  the  fleet  recapitalization,  and  cost,  schedule,  and 
perfonnance  trades  may  be  made.  Navy  AoAs  can  address  the  fleet  needs,  because 
they  are  so  closely  related  to  the  materiel  solution  (i.e.,  ships).  For  the  Anny,  it  is 
different.  The  Anny  has  portfolio  strategies  and  vehicle  modernization  strategies. 
Anny  AoAs  don’t  address  these  portfolio  and  vehicle  modernization  issues.  The 
organizations  perfonning  the  AoAs,  TRAC  for  example,  don’t  work  on  these  issues. 
These  issues  are  in  the  realm  of  CAA  and  Headquarters  Department  of  the  Anny 
(HQDA). 

•  The  AoA  should  assure  the  system  aligns  with  the  new  National  Defense  Strategy — 
OSD  CAPE  is  interested  in  this. 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

The  definition  of  the  alternatives  should  be  tailored  to  the  capabilities  of  the  system. 

•  The  definition  of  the  alternatives  should  be  tailored  to  the  capabilities  of  the  system. 
The  alternatives  should  be  tailored  to  the  situation. 
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Analyses  should  be  conducted  before  MDD  to  determine  alternatives. 

•  There  should  be  a  set  of  analyses  before  MDD  to  determine  the  alternatives.  OSD  had 
a  MDD  Decision  Brief  Template  that  includes  14  issues  to  be  covered. 

Screening  analysis  is  beneficial  to  detennine  the  alternatives. 

•  Screening  analysis  is  helpful  to  detennine  the  alternatives. 

•  Screening  analysis  of  many  alternative  candidates  is  very  helpful. 

Cost,  schedule,  and  performance  trades  should  be  conducted  within  the  alternatives. 

•  WSARA  discusses  cost,  schedule,  and  performance  trades.  The  trades  can  be  done 
within  alternatives,  but  AoAs  need  to  look  at  trades  intelligently  with  respect  to  cost, 
schedule,  and  perfonnance  depending  on  the  maturity  of  our  understanding  of  the 
capability.  The  Army  does  this  by  taking  examples  or  cases  (alternatives)  and  making 
cost,  schedule,  and  performance  trades. 

Parametric  analysis  should  be  perfonned  to  detennine  the  most  important  attributes. 

•  Parametric  analysis  of  attributes  should  be  performed  to  determine  the  set  of  most 
important  attributes  that  drive  capability.  In  turn,  this  supports  development  of  more 
accurate  cost  and  affordability  estimates. 

•  Affordability  must  cause  us  to  look  hard  at  reducing  attribute  values.  An  analogy  is 
purchasing  an  automobile.  The  buyer  of  an  automobile  identifies  a  set  of  needed 
attributes  for  a  car  (e.g.,  manual  transmission,  30  miles  per  gallon,  moderate 
acceleration  capability,  and  trunk  space).  There  also  are  numerous  desired  attributes. 
The  affordability  aspect  would  cut  down  on  the  desired  attributes.  A  Volkswagen 
may  have  to  be  selected  instead  of  an  Audi. 

AoAs  should  compare  cases  rather  than  alternatives.  Cases  are  examples  of  alternatives. 

•  There  is  a  tenninology  issue  associated  with  alternatives  in  AoAs.  The  comparison  in 
AoAs  is  made  of  cases.  These  cases  are  representative  examples  of  alternatives.  They 
should  not  be  called  alternatives.  For  example,  there  are  cost  and  performance 
representative  examples  of  NDI.  Cost,  schedule,  and  perfonnance  associated  with 
these  systems  can  be  compared  to  other  cases.  These  NDI  cases  should  not  modify 
the  foreign  systems. 
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Alternatives  represent  “stalking  horses”  to  identify  an  achievable,  affordable,  and 
operationally  relevant  set  of  attributes.  The  purpose  of  having  alternatives  is  to  identify  these 
attributes. 

•  Alternatives  are  the  “stalking  horses”  to  get  at  an  achievable,  affordable,  and 
operationally  relevant  set  of  attributes.  There  always  is  a  baseline  alternative,  which 
currently  the  Anny  has.  The  other  alternatives  represent  groups  of  attributes.  The 
AoA  is  about  selecting  a  set  of  attributes  that  meet  the  three  fundamental  criteria 
(affordable,  achievable,  and  operationally  relevant),  instead  of  selecting  a  specific 
alternative. 

The  Anny  should  work  closely  with  OSD  CAPE  to  obtain  agreement  on  the  alternatives. 

•  Alternatives  should  include  a  baseline  plus  other  acceptable  alternatives.  These 
alternatives  should  be  developed  collectively  by  the  AoA  team  and  with  buy-in  from 
the  SAG.  Recent  AoAs  have  shown  varied  degrees  of  getting  buy-in  at  the  SAG. 
Murder  boards  should  be  conducted  as  well  as  developing  systems  books.  These 
would  include  defining  key  capabilities  and  getting  agreement  of  alternatives  set  by 
all  members  of  the  SAG. 

•  The  alternatives  should  be  locked  in  with  OSD.  They  should  be  clearly  defined  and 
locked  in  before  much  time  and  effort  is  spend  analyzing  these  alternatives. 

•  The  Army  must  get  OSD  CAPE  to  participate  more  and  obtain  the  people  with  the 
right  skill  sets.  The  Anny  can’t  keep  the  AoA  internal,  and  they  must  let  in  outsiders. 

6,  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

Foreign  systems  should  be  considered  as  equivalent  to  U.S.  systems  with  the  same 
fidelity  of  data. 

•  Foreign  systems  should  be  considered  equivalent  to  other  systems  in  AoAs. 

•  There  should  be  the  same  fidelity  of  data  for  the  foreign  systems,  so  there  is  not  an 
“apples-and-oranges”  comparison  based  on  data.  If  the  foreign  systems  do  not  have 
the  same  level  of  data,  it  would  be  difficult  to  compare  the  alternatives. 

•  Viable  foreign  system  alternatives  should  be  included  in  AoA. 

Foreign  systems  represent  NDI  cases.  Therefore,  they  should  not  be  modified. 

•  Foreign  systems  are  representative  of  NDI  cases.  They  have  a  cost  and  performance 
associated  with  them.  There  is  cost,  schedule,  and  performance  associated  with  these 
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systems  that  can  be  compared  to  other  cases.  These  NDI  cases  should  not  modify  the 
foreign  systems. 

The  United  States  may  not  purchase  a  foreign  system  for  statutory  reasons. 

•  The  question  is  :  Why  should  foreign  system  be  considered  in  AoAs?  There  are  laws 
that  preclude  the  United  States  from  buying  foreign  systems.  Foreign  system 
manufacturers  may  go  into  business  with  U.S.  companies  to  sell  the  system. 

•  Even  foreign-developed  systems  would  be  manufactured  in  the  United  States.  The 
United  States  would  have  to  be  willing  to  buy  the  foreign  system,  but  there  is  no 
political  will  at  this  point. 

Foreign  systems  should  only  be  considered  in  AoAs  for  comparison  purposes  only  to 
investigate  capabilities  and  attributes. 

•  Unless  the  United  States  is  seriously  considering  buying  the  foreign  system,  having  a 
foreign  system  in  the  AoA  is  for  comparison  purposes  only.  Analyzing  the  foreign 
system  would  be  beneficial  to  examine  its  capabilities  and  determine  if  they  would  be 
useful  on  the  U.S.  system. 

•  Foreign  systems  should  only  be  included  as  an  exception  and  only  to  help  identify  a 
set  of  attributes. 

•  Foreign  systems  would  be  good  for  competition.  For  the  “concept  car,”  foreign 
companies  could  be  invited. 

Foreign  systems  will  have  to  be  modified  to  meet  U.S.  standards. 

•  At  a  minimum,  the  foreign  system  will  have  to  be  changed  to  adapt  to  U.S.  military 
standard. 

Foreign  systems  should  be  discussed  with  OSD  CAPE  early  in  the  AoA  process  to 
detennine  how  they  will  be  considered. 

•  Foreign  system  alternatives  should  be  discussed  early  with  OSD  CAPE.  The  AoA 
team  must  clearly  define  how  they  are  considering  the  foreign  systems  in  the  AoA. 
This  discussion  of  foreign  systems  will  help  set  the  parameters  of  the  analysis. 

A  “concept  vehicle”  approach  to  developing  systems  could  enhance  AoAs. 

•  Having  a  “concept  vehicle”  would  allow  more  data  to  be  collected  and  a  better 
comparison  made.  Developing  concept  vehicles  early  in  the  process  will  allow 
perfonnance  data  to  be  collected.  These  concept  vehicles  can  be  compared  to  the 
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current  combat  vehicles.  The  big  advantage  would  be  that  the  Army  is  not  committed 
to  buying  any  of  the  concept  vehicles  into  a  full  production  model.  The  Anny  only 
buys  the  concept  vehicles.  If  the  concept  vehicles  look  promising,  the  Army  can 
invest  more  money  in  conducting  an  AoA.  If  the  AoA  concludes  the  concept  vehicles 
are  better  than  the  current  vehicles  and  that  the  cost  to  replace  the  current  vehicle  is 
acceptable,  then  the  Army  can  develop  prototypes  for  competition. 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs? 

Tradeoff  analyses  should  be  tailored  to  the  capability,  maturity,  and  cost  of  the  system. 

•  It  depends  on  the  capability,  maturity,  and  cost  of  the  systems. 

Trades  should  be  performed  within  and  among  alternatives  to  address  the  decision  space. 

•  For  early  in  the  process  of  defining  alternatives,  the  cost,  schedule,  and  performance 
trades  should  be  done  internally  within  the  alternative.  Cost,  schedule,  and 
perfonnance  trades  should  be  considered  in  the  AoA  prior  to  MS  A  and  post  MS  A. 
Sensitivity  analysis  should  be  conducted  to  address  the  among-alternative  trades  (i.e., 
the  inter-alternative  trades).  Both  the  within  and  among  trades  have  to  be  performed 
to  address  the  decision  space. 

The  tradeoff  analysis  will  support  the  requirements  development  process,  including  the 
needed  attributes. 

•  The  trades  should  be  associated  with  requirements.  The  requirements  drive  the  cost, 
schedule,  and  perfonnance.  For  schedule,  the  requirements  drive  the  TRL  and  the 
time  to  mature  the  technologies.  This  maturation  time  is  a  factor  in  the  schedule 
length.  The  “Cost-Informed-Trades”  effort  is  along  the  lines  of  what  WSARA  is 
looking  for. 

•  The  trades  have  to  arrive  at  the  minimum  needs. 

•  The  classic  purpose  of  an  AoA,  at  the  end  of  the  day,  is  to  determine  which  attributes 
are  important. 

Risk  assessment  should  be  performed  on  tradeoffs  between  cost,  schedule,  and 
perfonnance. 

•  Risk  assessment  can  be  done  on  the  effects  of  the  trades. 
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•  Tradeoffs  of  cost,  schedule,  and  perfonnance  should  be  addressed  in  AoAs  in  terms 
of  risk — what  is  risk  to  cost,  schedule,  and  perfonnance  ?  To  add  value,  AMSAA  is 
perfonning  risk  assessments  in  AoAs.  Detennining  if  technologies  are  not  feasible  in 
the  time  frame  is  important. 

There  were  differing  priority  of  factors — including  cost,  schedule,  and  perfonnance;  and 
perfonnance,  cost,  and  schedule. 

•  Cost,  schedule,  and  performance  should  all  be  considered.  Cost  is  the  first  among 
equals — an  affordable  alternative  is  paramount.  Cost  targets,  in  tenns  of  APUC, 
AMUC  or  LCC  should  be  identified  (APUC  is  the  best).  After  the  cost  targets  are 
identified,  the  other  two  items  (schedule  and  performance)  are  traded.  Schedule  is 
next  in  importance.  The  question  is:  What  is  the  Anny  willing  to  accept  to  get  the 
system  in  the  hands  of  the  soldier  quicker?  These  tradeoffs  must  be  laid  out 
analytically. 

•  Technical/performance  ought  to  be  at  the  top  of  the  list  when  trading  cost,  schedule, 
and  performance.  Cost  is  becoming  increasingly  important.  Schedule,  while 
important,  may  be  a  lesser  priority  than  the  others,  because  of  uncertain  thresholds 
and  needs. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

AoAs  must  address  the  decision  process  and  provide  utility  to  the  decision-makers. 

•  The  AoAs  must  be  tied  to  the  decision  process  that  matters. 

•  AoAs  must  do  more  than  offer  up  results  that  the  community  intuitively  knows.  AoAs 
must  provide  more  utility  to  decision-makers. 

AoAs  must  satisfy  all  the  customers  including  OSD  CAPE,  the  acquisition  community, 
the  requirements  community,  and  the  portfolio  mix  community. 

•  The  AoA  must  satisfy  all  the  customers  including  OSD  CAPE,  acquisition 
community,  the  requirements  community,  and  the  portfolio  mix  community. 
Knowledge  points  would  help  integrate  AoAs  and  provide  Anny  and  OSD  senior 
leaders  better  infonnation. 

AoAs  must  address  OSD  CAPE  guidance  and  issues. 

•  Continue  to  ensure  that  the  AoA  team  effectively  addresses  all  OSD  CAPE  issues 
documented  in  the  AoA  guidance  and  identified  during  SAG  meetings. 
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•  Also,  continue  to  ensure  that  results  of  Anny  analyses  and  results  regarding  OSD 
CAPE  issues  are  documented  in  the  AoA  report. 

The  Anny  must  engage  OSD  CAPE  in  earlier  in  the  AoA  process.  The  Army  must  be 
candid  and  open  with  OSD  CAPE. 

•  One  key  is  to  bring  OSD  CAPE  in  earlier  and  to  get  their  buy-in  early.  The  Army  has 
to  be  candid  with  OSD  CAPE.  To  reduce  the  number  of  questions  OSD  CAPE  asks 
later  in  the  AoA  process,  the  Army  should  work  with  OSD  CAPE  to  lock  in  the 
defining  problems.  The  SAG  should  be  used  to  provide  azimuth  checks  to  make  sure 
the  Army  is  where  it  ought  to  be. 

The  AoA  guidance  should  only  include  three  points:  (1)  the  purpose  for  the  AoA;  (2) 
when  the  AoA  results  are  needed;  and  (3)  identify  the  key  angst/friction/concern  points  or  issues. 

•  Too  much  time  is  spent  framing  the  problem.  The  process  of  framing  the  problem  is 
too  bureaucratic  and  backward.  The  TRADOC  COEs  know  when  they  are  developing 
the  CDD,  which  must  serve  as  a  trigger  to  initiate  the  AoA.  ARCIC,  armed  with  the 
when  and  why  (the  gap  to  satisfied),  should  communicate  with  Army  staff  (e.g.,  G3 
and  G8),  as  well  as  OSD  AT&L  and  OSD  CAPE.  There  must  be  an  enforced  timeline 
of  approximately  30  days  to  frame  the  problem  and  publish  the  directive.  The 
directive  allows  the  study  agency  access  to  other  organizations  to  help  with  the  study. 
The  AoA  directive,  or  guidance,  should  only  include  three  things:  (1)  the  purpose  for 
the  AoA  (why,  typically,  a  MS  designation);  (2)  when  the  AoA  results  are  needed;  (3) 
identify  the  key  angst/friction/concern  points  or  issues.  The  directive/guidance  should 
not  consider  other  areas  that  are  typically  included.  Everyone  would  benefit  with  only 
these  three  points  in  the  directive/guidance. 

•  The  Anny  must  do  more  to  drive  OSD  CAPE  to  provide  AoA  guidance  much  earlier 
in  the  process. 

Anny  requirements  must  be  developed  with  the  support  of  analyses. 

•  Requirements  are  being  developed  without  analysis.  The  JCIDS  process  is  too  vague. 
The  PMs  have  great  latitude  of  what  to  include  in  the  system  specs.  Sometimes  there 
are  interactions  between  the  user  and  the  PM  independent  of  the  AoAs.  This  may 
cause  internal  disagreements. 

The  Anny  needs  more  analytic  capability  to  address  AoAs. 
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•  The  Army  needs  to  build  more  analysis  capability,  especially  within  COE 
Capabilities,  Development  and  Integration  Directorates  (CDIDs).  They  could  perfonn 
more  analysis  instead  of  TRAC,  or  add  depth  to  TRAC-led  AoA  efforts.  The  Anny 
also  has  to  build  analysis  in  the  cost  area,  especially  in  TRADOC.  Risk  assessment 
methodology  must  be  codified  in  AoA  policy  and  regulatory  guidance  and  directives. 

•  The  Army  should  spend  more  time  defining  parameters,  checking  assumptions,  and 
examining  emerging  data. 

The  Army  could  utilize  a  “concept  vehicle  show”  for  industry  to  demonstrate  their 
technologies. 

•  The  Anny  can  learn  a  lot  from  the  process  industry  uses  in  developing  and  marketing 
new  vehicles.  The  Army  should  host  an  annual  “Concept  Vehicle  Show”  similar  to 
the  “Detroit  Auto  Show.”  The  Army  would  need  to  spend  RDT&E  money  to  fund  the 
concept  vehicles.  The  concept  vehicles  would  need  to  be  actual  replicas  of  a  combat 
vehicle  system,  so  the  Army  could  collect  perfonnance  data.  These  data  could  be 
used  to  develop  a  “systems  book”  for  modeling  the  concept  vehicle  in  the  Anny’s 
combat  models.  There  would  be  no  obligation  for  the  Army  to  buy  any  of  the  concept 
vehicles  it  develops.  The  purpose  of  the  annual  event  is  to  allow  industry  to 
demonstrate  what  it  could  produce  in  the  near  future.  The  Anny  could  use  the  data  to 
compare  the  concept  vehicle  against  the  current  set  of  vehicles  to  detennine  whether 
buying  the  new  system  is  appropriate.  It  may  be  better  to  keep  the  new  systems  “on 
the  shelf’  for  the  future.  If  the  AoA  concludes  the  concept  vehicles  are  better  and  the 
cost  is  reasonable,  the  Army  can  release  a  Request  for  Proposal  (RFP)  for  industry  to 
develop  competitive  prototypes  for  source  selection.  This  may,  or  may  not,  add  time 
to  the  acquisition  process.  The  answer  might  depend  on  when  you  start  the  clock.  If 
you  assume  the  Anny  will  always  host  an  Annual  Concept  Vehicle  Show,  after  3  to  5 
years,  the  Anny  should  have  plenty  of  concept  vehicles  from  which  to  use  as  a 
starting  point,  which  should  shorten  the  overall  length  of  getting  a  program  through 
the  acquisition  process.  If  there  are  no  concept  vehicles  to  use  a  starting  point, 
acquisition  timeline  would  be  longer. 
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APPENDIX  C 

OSD  CAPE  INTERVIEW  RESULTS 

This  appendix  provides  the  answers  for  the  OSD  CAPE  interviewees  (S.  Conies  &  E. 
Simmons,  personal  communication,  April  1,  2012;  K.  Kaspersen,  personal  communication, 
February  21,  2012;  J.  Duke  &  E.  Fulmer,  personal  communication,  February  24,  2012;  C.  Allen, 
personal  communication,  March  25,  2012).  For  each  question,  the  interviewees’  answers  are 
grouped  by  general  ideas.  A  summary  of  the  key  points  for  all  interviewees  is  provided  followed 
by  the  interviewees’  specific  comments  as  bullet  points. 

1.  Briefly  explain  in  your  own  words  the  purpose  of  an  AoA. 

There  are  numerous  purposes  to  AoA,  including  providing  infonnation  to  inform 
acquisition  decisions,  exploring  the  trade  space  of  cost  schedule  and  performance,  and  answering 
the  “Why”  questions. 

•  To  ensure  that  leadership  has  the  information  to  make  acquisition  decisions. 

•  To  inform  acquisition  decisions. 

•  Providing  infonnation  for  making  investment  decisions. 

•  To  answer  the  question  “why”  Is  it  the  right  time  to  acquire  the  materiel  solution? 
Does  the  materiel  solution  meet  the  right  purpose  and  fill  the  capability  gap? 

•  Exploring  of  the  trade  space  between  cost,  schedule,  and  perfonnance. 

There  are  desired  characteristics  of  the  options  and  alternatives  including:  objectivity, 
nonpreferential  options,  affordable  alternatives,  and  a  robust  set  of  alternatives. 

•  The  AoA  should  be  objective  without  a  prefened  option. 

•  The  AoA  should  provide  feasible  alternatives  utilizing  an  upfront  screening  analysis 
to  ensure  they  are  affordable  and  to  mitigate  the  capability  gaps.  A  robust  set  of 
alternatives  should  be  considered.  If  two  or  three  alternatives  are  not  included,  the 
analysis  may  be  questionable. 

•  The  AoA  provides  a  range  of  options  and  the  cost,  schedule,  and  perfonnance  of  each 
option. 
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The  analyses  should  back  up  a  recommended  solution,  show  due  diligence,  explain  the 
trade  space,  include  sensitivity  analysis,  and  consider  the  positive  and  negative  aspects  of  each 
option. 

•  The  analysis  must  back  up  a  recommended  solution.  The  AoA  should  answer  why  a 
particular  solution  is  recommended. 

•  The  AoA  has  to  show  due  diligence  in  the  analysis  of  the  options  and  alternatives. 

•  The  AoA  does  not  have  to  produce  a  clear  answer,  but  options  with  different  costs 
and  development  schedules. 

•  The  AoA  should  explain  the  trade  space.  The  AoA  should  provide  trades  within  and 
across  alternatives  among  cost,  schedule,  and  perfonnance.  “Within”  alternatives 
include  subsystem  trades  for  differences  in  capability  and  “across”  includes  variations 
at  a  system  level. 

•  The  AoA  should  include  a  sensitivity  analysis  to  help  determine  importance.  This 
analysis  should  identify  cost  and  performance  drivers. 

•  The  positives  and  negatives  of  each  option  should  be  explored. 

The  results  of  the  AoAs  should  inform  requirements,  identify  cost-effective  solutions,  be 
transparent,  and  consider  operational  effectiveness. 

•  The  AoA  should  inform  requirements.  Although  the  goals  may  be  the  CDD  values, 
sensitivity  analyses  should  determine  a  decrease  in  the  performance  by  10  percent  and 
detennine  the  resulting  cost  reduction. 

•  Requirements  cost  must  be  identified,  and  the  JROC  must  estimate  how  much  the 
system  will  cost  based  on  their  requirements.  The  most  cost-effective  solutions  should 
be  identified. 

•  The  AoA  process  should  be  transparent  and  include  a  rigorous  and  complete  analysis 
of  the  alternatives.  OSD  CAPE  requires  an  understanding  of  the  analyses. 

•  The  quality  of  the  end  results  should  include  an  impact  on  operational  effectiveness 
metrics — this  analysis  should  include  results  from  a  combined  arms  operational 
simulation. 

•  AoAs  should  include  graphs  that  plot  blue  effectiveness  with  cost.  Graphs  could 
include  separate  effects  for  their  cost. 
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2.  What  do  you  think  are  the  major  issues  associated  with  AoAs  under  the  new  WSARA 
guidance? 

OSD  CAPE  has  an  increased  level  of  responsibility  for  AoAs  under  WSARA,  that 
includes  the  following:  issuing  AoA  guidance,  determining  if  the  AoA  was  performed  consistent 
with  OSD  CAPE’s  guidance,  and  recommending  AoA  updates. 

•  Under  WSARA,  OSD  CAPE  provides  the  guidance  and  also  detennines  whether  the 
AoA  was  done  in  accordance  with  the  guidance. 

•  Before  WSARA,  USD(AT&L)  sent  out  the  AoA  guidance,  although  OSD  CAPE 
developed  this  guidance.  After  WSARA,  OSD  CAPE  develops  and  issues  the 
guidance  for  AoAs.  The  change  has  allowed  OSD  CAPE  more  influence  in  the  AoA 
guidance. 

•  AoAs  have  to  be  prepared  consistently  with  OSD  CAPE  guidance,  and  CAPE  has  to 
certify  that  the  AoA  was  prepared  consistently  with  OSD  CAPE  guidance. 

•  OSD  CAPE  has  more  responsibilities  for  AoAs.  It  required  a  dynamic  AoA  update 
for  a  recent  Army  program. 

There  is  a  new  requirement  under  WSARA  to  perfonn  cost,  schedule,  and  performance 
trades.  AoAs  must  ensure  that  the  trade  space  is  explored  in  detail. 

•  WSARA  includes  language  requiring  a  comprehensive  look  at  the  trades  between 
cost,  schedule,  and  performance. 

•  A  major  change  in  AoA  is  explaining  and  documenting  the  trade  space  between  cost, 
schedule,  and  performance.  Decision-makers  should  have  a  better  understanding  of 
the  underlying  problems  with  cost,  schedule,  and  perfonnance.  AoAs  must  ensure 
that  the  trade  space  is  explored  in  detail.  Circumstances  change,  and  there  may  be  a 
different  “lever  that  is  more  suitable  to  pull.”  There  may  be  more  flexibility  to  change 
a  new  system  than  an  existing  system. 

WSARA  has  increased  the  visibility  of  AoAs,  which  has  resulted  in  a  greater  awareness 
and  interest  in  AoA  results.  The  JS,  the  Services,  Congress,  and  GAO  are  inquiring  about  the 
results  of  AoAs. 
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3.  How  do  you  think  AoAs  should  be  tailored  for  different  MSs? 

An  AoA  is  required  at  MS  A  and  there  is  one  set  of  guidance,  which  is  good  for  all  MS 
decisions.  The  guidance  will  supply  questions  to  be  answered,  and  the  AoA  team  should  answer 
the  questions  to  their  best  ability  with  the  information  available. 

•  There  is  only  one  AoA  required  and  one  set  of  AoA  guidance.  The  guidance  is  to  be 
issued  at  the  MDD. 

•  OSD  CAPE  provides  one  set  of  AoA  guidance  for  the  acquisition  effort,  which  is 
good  for  all  MS  decisions.  The  guidance  will  supply  questions  to  be  answered.  The 
AoA  guidance  is  due  at  MDD,  but  OSD  CAPE  may  not  know  where  the  DAE  will 
start  the  program  in  the  acquisition  process.  The  AoA  team  should  answer  the 
questions  to  its  best  ability  with  the  infonnation  available.  The  AoA  team  may  not  be 
able  to  answer  all  questions,  but  it  must  document  why  it  cannot  answer  the 
questions. 

An  AoA  does  not  have  to  be  updated  for  the  subsequent  MSs  unless  OSD  CAPE 
designates  an  update  is  required.  An  AoA  may  have  to  be  updated  for  the  following  reasons:  (1) 
the  underlying  assumption/facts  change,  (2)  a  new  threat  is  identified,  and  (3)  the  program 
changes  acquisition  categories  that  change  the  MDA  (e.g.,  from  the  AAE  to  the  DAE). 

•  The  AoA  does  not  have  to  be  updated  for  the  subsequent  MSs  unless  OSD  CAPE 
designates  an  update  is  required.  Certain  factors  would  require  an  AoA  update.  If  an 
updated  AoA  is  not  required,  the  Services  still  would  conduct  infonnal  analysis  to 
support  the  acquisition  process.  A  fonnal  AoA  may  not  be  required. 

•  OSD  CAPE  may  request  an  update  of  the  AoA  to  answer  the  questions  that  could  not 
be  answered  in  the  original  AoA. 

•  An  AoA  may  have  to  be  updated  for  the  following  reasons:  (1)  the  underlying 
assumption/facts  change,  (2)  a  new  threat  is  identified,  and  (3)  the  program  changes 
ACATs  that  change  the  MDA  (e.g.,  from  the  AAE  to  the  DAE). 

•  Reasons  for  updating  an  AoA  would  include  a  threat  change,  affordability  or  cost 
estimates  change,  the  requirement  change,  performance  change,  or  some  other  change 
that  invalidates  the  analysis.  There  are  no  set  percentages  associated  with  these 
changes  to  require  an  update  on  the  AoA.  On  a  case-by-case  basis,  the  detennination 
is  made  whether  an  AoA  update  is  required. 
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•  An  update  of  the  MS  A  AoA  may  be  required  if  the  threat  changes  or  the  technology 
changes. 

AoAs  may  be  required  at  later  MS  decisions  for  (1)  program  going  directly  from  MDD  to 
MS  B  or  post-MS  B  or  (2)  a  program  that  changes  ACAT  levels. 

•  If  the  MDA  detennines  that  the  program  will  go  directly  to  MS  B,  there  usually  are 
fewer  alternatives,  better  confidence  in  the  cost  estimates  (should  have  APB  cost), 
more  real-world  data  available  (prototype  testing),  and  more  information  on 
developmental  risk. 

•  For  MS  A,  the  alternatives  are  not  as  firm  as  in  AoAs  conducted  at  later  MSs.  There 
will  be  more  detail  if  a  MS  B  AoA  is  required. 

•  There  could  be  instances  where  an  ACAT  II  program  increases  in  scope  to  an  ACAT 
I  program  in  later  MSs.  The  movement  of  the  program  to  an  ACAT  I  status  would 
require  an  AoA  to  be  perfonned.  AoA-like  analyses  should  have  been  perfonned  for 
the  earlier  MSs  which  could  be  used  in  the  fonnal  AoA. 

AoAs  should  be  tailored  to  the  specific  program  rather  than  for  the  MS.  The  analyses  are 
tailored  for  the  particular  program  rather  than  for  the  MS. 

4.  What  types  of  analysis  do  you  think  need  to  be  performed  in  AoAs  to  meet  OSD  CAPE 
expectations? 

The  analysis  should  be  quantitative  instead  of  qualitative  judgment-based.  Using  Subject 
Matter  Experts  (SMEs)  to  detennine  how  much  they  desire  a  particular  attribute,  scoring  the 
results,  and  developing  weighing  factors  are  not  the  correct  process. 

•  Analyses  should  be  quantifiable  instead  of  qualitative  judgment-based  analyses  (i.e., 
value-focused  thinking).  Using  SMEs  to  detennine  how  much  they  desire  a  particular 
attribute,  scoring  the  results,  and  developing  weighing  factors  are  not  the  correct 
process.  Making  these  qualitative  results  quantitative  also  is  not  appropriate. 

•  The  analyses  should  be  quantifiable  and  based  on  engineering  and  test  data.  For  a 
recent  AoA,  various  alternatives  were  operated,  and  the  data  were  recorded. 

The  analyses  must  be  understandable,  transparent,  and  support  the  conclusion. 

•  The  analyses  should  be  understandable,  transparent,  compare  alternatives,  provide 
rationale  and  support  a  conclusion.  The  quality  of  the  analyses  should  be  high. 
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•  The  analyses  should  be  transparent.  OSD  CAPE  should  not  have  to  conduct  forensics 
to  determine  how  the  analyses  were  performed. 

Exploratory  analysis  and  cost,  schedule,  and  perfonnance  trades  are  important 
components  to  an  AoA. 

•  The  analyses  should  be  exploratory. 

•  There  should  be  a  trades  analysis  of  cost,  schedule,  and  perfonnance.  These  trades 
should  be  within  and  across. 

•  The  analysis  should  provide  the  basic  knowledge  for  the  decision-maker  with  the 
trade  space  explored. 

Sensitivity  analyses  should  be  performed  for  the  assumptions,  alternative  definitions,  and 
drivers.  For  the  “lesser”  alternatives,  sensitivity  analyses  should  be  performed  to  detennine  what 
it  would  take  to  make  it  a  more  competitive  alternative. 

•  Analyses  should  be  performed  to  detennine  the  sensitivity  of  the  results  to  changes  in 
the  assumptions.  The  AoA  should  identify  where  the  assumptions  degrade  and  when 
this  degradation  matters  or  does  not  matter. 

•  For  the  “lesser”  alternatives,  sensitivity  analyses  should  be  performed  to  determine 
what  it  would  take  to  make  it  a  more  competitive  alternative.  The  AoA  should 
address  potential  ways  a  lesser  alternative  could  be  improved.  The  sensitivity  should 
address  an  increase  and  decrease  in  performance. 

•  Sensitivity  analysis  also  should  be  perfonned  to  detennine  the  sensitivity  to  drivers. 

•  Caution  must  be  used  for  selecting  scenarios.  A  scenario  may  drive  the  answer. 

The  analyses  should  identify  “tipping  points”  for  the  decision  choices.  If  there  are  no 
tipping  points,  or  points  of  interest,  the  AoA  should  “stop  drilling”  when  the  result  does  not 
make  a  distinction  between  alternatives. 

•  The  analyses  should  find  the  tipping  points  and  points  of  interest. 

•  If  there  are  no  tipping  points,  or  points  of  interest,  the  AoA  should  “stop  drilling” 
when  the  result  does  not  make  a  distinction  between  alternatives. 

•  If  an  analysis  makes  no  distinction,  OSD  CAPE  does  not  want  research  for  academic 
reasons.  The  Service  should  go  back  to  SAG  and  cease  work  on  that  analysis.  Also, 
the  AoA  should  not  map  out  nonviable  solutions. 
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The  following  analyses  should  be  conducted  in  AoAs:  operational  effectiveness, 
operational  assessments,  performance,  cost,  sensitivity,  fully  burdened  cost  of  fuel,  and 
affordability/portfolio. 

•  The  basic  types  of  analysis  should  be  conducted,  such  as  cost,  performance,  mission 
effectiveness,  sensitivity,  and  fully  burdened  cost  of  fuel. 

•  The  specific  types  of  analyses  perfonned  for  AoAs  will  depend  on  the  program. 
Analyses  should  be  tailored  to  the  particular  program.  In  some  cases,  systems  level 
analysis  (i.e.,  engineering  level  analysis)  may  be  good  enough.  There  should  be  an 
operational  effectiveness  analysis  and  a  comparison  of  operational  effectiveness  vs. 
cost.  The  AoA  has  to  put  the  system  in  the  war  fight. 

•  Technical  perfonnance  analysis  should  be  included  in  the  AoA,  which  includes 
measures  and  metrics.  An  operational  assessment  should  be  included  to  estimate  how 
effective  the  alternatives  are.  In  some  cases,  this  operational  assessment  is  a  map 
exercise  or  a  tabletop  exercise  with  an  After  Action  Report. 

•  Cost  must  be  put  into  the  context  of  affordability  (i.e.,  portfolio  analysis). 

•  Capability  metrics  can  be  used  to  determine  if  the  alternative  is  better  than  the 
baseline.  Specifications  are  too  specific. 

•  Non-materiel  solutions  should  still  be  considered. 

5.  How  do  you  think  definitions  of  the  alternatives  should  be  developed? 

The  basic  alternatives  for  AoAs  include  a  baseline  of  what  is  currently  programmed  in 
service,  a  modification  of  what  is  currently  in  service,  an  NDI  system/foreign  system,  and  a  new 
start  system. 

•  The  definitions  for  the  basic  alternatives  are  usually  clear  and  include  the  following: 
what  you  have  in  the  inventory  today,  a  modification  of  what  is  available  in  the 
inventory  today,  a  new  system,  or  something  else  that  is  available. 

•  One  alternative  should  be  the  program  or  system  that  is  funded  today  (i.e.,  the 
baseline).  A  foreign  alternatives  that  also  exists  today  should  be  included,  if  feasible. 

•  There  should  be  a  baseline  of  what  is  currently  programmed  for  the  Service.  The 
other  alternatives  should  have  better  capability  based  on  the  gaps.  Another  alternative 
should  be  a  modified  baseline.  A  new  start  can  be  another  alternative.  There  should 
also  be  an  NDI  system. 
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There  should  be  a  broad  search  for  alternatives,  and  the  definition  of  those  alternatives 
should  be  kept  broad.  The  Service  should  not  presuppose  an  answer  and  prematurely  reduce  the 
decision  space. 

•  The  alternatives  should  be  general,  and  they  should  not  be  overly  prescriptive.  The 
Service  should  not  presuppose  an  answer  and  close  the  aperture  too  quickly.  If  an 
answer  assumed  is  too  quickly,  the  Service  may  be  left  with  no  options  if  that  answer 
is  not  feasible.  The  Services  should  not  to  prematurely  reduce  the  decision  space.  The 
search  should  be  kept  broad  until  the  information  dictates  that  the  search  converges. 

A  robust  screening  analysis  should  be  completed  up  front  and  presented  to  the  SAG, 
which  will  request  additional  excursions  and  reframe  or  expand  the  analysis,  as  necessary. 

•  A  robust  screening  analysis  should  be  completed  upfront.  Initial  results  of  AoA  will 
be  presented  to  the  SAG,  which  is  led  by  OSD  CAPE.  The  SAG  will  request 
additional  excursions,  and  reframe  or  expand  the  analysis,  as  necessary. 

The  alternatives  should  cover  the  decision  space,  and  trades  and  sensitivity  analyses 
should  be  performed.  A  good  screening  analysis  would  help  explore  the  trade  space. 

•  The  alternatives  should  cover  the  decision  space,  including  the  end  points  and  some 
points  in  between.  End  points  would  include  the  baseline  and  a  new  start.  There  should 
be  not  only  two  alternatives,  but  some  points  in  between  the  baseline  and  a  new  start. 
These  in-between  points  would  help  explore  the  trade  space.  In  addition  to  the 
alternatives,  the  AoA  must  look  at  trades  within  the  possible  alternatives. 

•  A  good  screening  analysis  would  help  explore  the  trade  space.  This  analysis  should 
identify  the  cost  and  performance  drivers,  and  detennine  the  alternatives. 

•  The  alternatives  should  provide  a  range  of  how  quickly  the  system  is  available. 

•  In  the  AoA,  variations  of  these  basic  alternatives  must  be  examined  in  sensitivity 
analyses. 

The  Services  should  follow  the  OSD  CAPE  study  guidance  when  developing  alternatives. 

•  The  Services  should  look  at  the  OSD  CAPE  study  guidance  to  develop  the 
alternatives.  The  alternatives  must  address  the  gaps  and  constraints. 

•  The  OSD  CAPE  guidance  provides  infonnation  on  the  alternatives. 
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6.  How  do  you  think  foreign  system  alternatives  should  be  considered  in  AoAs? 

Foreign  systems  should  be  considered  in  AoAs,  and  the  Services  should  do  their  best  to 
obtain  data  and  infonnation.  Systems  from  threat  countries  should  not  be  considered. 

•  Foreign  alternatives  should  be  considered  in  AoAs. 

•  The  consideration  of  foreign  alternatives  is  difficult.  The  Services  should  do  their  best 
to  investigate  the  foreign  alternatives.  They  can  obtain  a  vehicle  or  visit  the  country. 

•  The  OSD  CAPE  guidance  would  include  foreign  systems.  How  the  foreign  systems 
are  considered  depends  on  the  specific  AoA  and  program. 

•  The  foreign  government  may  not  share  the  technology.  There  should  be  enough 
foreign  system  data  to  conduct  the  initial  screening.  It  is  questionable  if  enough  data 
would  be  available  to  perform  the  AoA,  but  it  is  the  SAG  decision  to  not  consider  the 
foreign  system  if  enough  data  are  not  available. 

•  Since  the  foreign  alternative  exists,  there  should  be  actual  test/fielding  data  available. 

•  Foreign  systems  should  stand  on  their  own  merit,  and  they  should  be  explored.  There 
are  some  countries  where  foreign  alternatives  should  not  be  considered  in  an  AoA. 
On  the  other  hand,  alternatives  should  not  be  thrown  out  just  because  they  are  foreign. 
The  Service  cannot  jump  to  only  U.S.  systems. 

•  Feasible  foreign  alternatives  should  be  considered  in  the  AoA,  which  includes 
systems  that  meet  the  capability  gaps  with  affordability.  Systems  from  some  countries 
may  not  be  feasible. 

The  decision  to  actually  purchase  a  foreign  alternative  is  not  OSD  CAPE’s  or  the  AoA 
team’s  decision. 

•  The  question  of  whether  DoD  actually  will  purchase  the  foreign  alternative  is 
someone  else’s  decision  besides  OSD  CAPE  or  the  AoA  team. 

The  cost  and  time  associated  with  modifying  foreign  alternatives  should  be  considered  in 
the  AoA. 

•  There  will  be  cost  and  time  associated  with  the  certification  and  modification  of  the 
foreign  systems  to  be  used  in  the  U.S.  military.  This  additional  cost  and  time  should 
be  considered  in  the  AoA.  The  AoA  should  identify  drawbacks. 

Another  purpose  of  considering  foreign  alternatives  is  to  learn  more  about  their  features, 
attributes,  technical  feasibility,  and  cost. 
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•  Foreign  systems  could  be  used  to  identify  desirable  features/attributes  that  can  be 
used  in  U.S.  systems. 

•  By  considering  the  foreign  alternatives,  the  Services  will  learn  about  technical 
feasibility,  performance,  or  cost. 

•  Foreign  alternatives,  if  selected,  would  be  fabricated  in  the  United  States.  Some  U.S. 
corporations  are  worldwide,  and  they  leverage  the  knowledge  of  the  foreign  systems 
or  build  the  foreign  systems. 

7.  How  do  you  think  the  tradeoffs  between  cost,  schedule,  and  performance  should  be 
considered  in  AoAs? 

The  tradeoff  analysis  should  explore  the  trade  space  and  decision  space  for  AoAs.  The 
boundaries  for  the  trade  space  should  be  expanded  to  ensure  a  robust  decision  space.  The  trades 
should  be  explored  throughout  the  AoA. 

•  The  tradeoff  analyses  should  explore  the  trade  space  between  cost,  schedule,  and 
perfonnance. 

•  The  AoA  should  show  the  implications  of  the  trades.  The  parameters  should  provide 
information  in  the  “infeasible”  zone.  For  example,  if  the  Service’s  schedule  limit  is  7 
years,  but  a  much  better  weapons  system  could  be  produced  in  8  years,  this  should  be 
identified  in  the  AoA. 

•  The  trades  should  not  be  locked  down  prematurely,  and  the  boundaries  should  be 
expanded.  OSD  CAPE  does  realize  there  analytical  capability  is  not  unlimited. 

•  The  trades  should  be  explored  throughout  the  AoA. 

•  For  a  good  tradeoff  analysis,  some  specific  attributes  should  be  varied.  The  biggest 
benefit  to  the  tradeoff  analysis  is  to  create  a  robust  decision  space  and  to  ensure 
completeness  of  the  analysis.  Some  attributes  may  be  varied  outside  the  practical 
limits  to  see  what  the  effect  would  be.  This  consideration  could  expand  the  decision 
space. 

AoAs  should  include  perfonnance  vs.  cost  curves. 

•  There  should  be  performance  vs.  cost  curves  developed  in  the  AoA.  The  tradeoff 
analysis  is  looking  for  the  knee  in  the  curve  of  performance  vs.  cost.  A  system  with 
the  best  perfonnance  but  a  very  high  cost  may  not  be  desired. 

The  priority  of  tradeoff  factors  are  performance,  schedule,  and  then  cost. 
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•  The  priority  of  the  tradeoff  factors  is  performance,  schedule,  and  then  cost.  The  AoA 
must  first  ensure  the  performance  is  worth  it.  The  AoA  must  determine  what 
perfonnance  level  can  be  obtained  for  each  option  and  then  detennine  if  this 
capability  is  important.  Schedule  is  the  second  priority.  Schedule  drives  cost. 
Optimistic  schedules  may  have  no  chance  of  succeeding,  and  the  additional  time 
required  would  increase  cost.  The  AoA  must  ensure  the  schedules  are  realistic.  Cost 
is  important,  but  not  as  important  as  the  other  two  factors. 

Operational  effectiveness  is  needed  in  the  tradeoff  analysis,  and  the  CAIG  should  be 
included  in  the  cost  estimates. 

•  The  independent  cost  organization  (CAIG)  should  be  involved  early  in  the  AoA 
process.  They  can  help  identify  the  real  cost,  and  whether  the  system  is  worth  the 
cost. 

•  The  effectiveness  of  the  alternatives  is  what  OSD  CAPE  desires. 

•  The  system  level  attributes  should  be  mapped  to  operational  effectiveness.  The 
tradeoff  analysis  should  include  operational  effectiveness  runs. 

AoA  should  inform  the  CDD;  although  the  CDD  should  not  be  written  to  make  one 
alternative  appear  better. 

•  AoAs  should  infonn  the  CDD;  although  the  CDD  should  not  be  written  for  the 
preferred  alternative  (the  preferred  alternative  going  into  an  AoA).  The  CDD  should 
not  be  changed  to  make  one  alternative  look  better. 

8.  What  would  you  do  differently  in  future  AoAs  to  better  meet  OSD  CAPE  expectations? 

OSD  CAPE  desires  more  transparency,  visibility,  understanding  of  the  analyses  and 
assumptions. 

•  OSD  CAPE  desires  better  visibility  into  the  AoA  and  a  better  understanding  of  the 
assumptions. 

•  OSD  CAPE  would  like  the  analysis  to  be  taken  up  a  level  (i.e.,  the  drivers  of  the 
results  should  be  explained).  More  visibility  should  be  provided  as  to  why  a  certain 
alternative  was  rated  “Red”  for  a  given  analysis.  The  AoA  should  identify  what  is 
driving  the  answers. 
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•  Insight  into  the  assumptions  is  critical.  Sometimes  the  AoA  team  makes  choices  that 
could  drive  the  assumptions  and  results. 

•  There  should  be  good  communication  between  the  Services  and  OSD  CAPE. 

•  The  analyses  have  to  be  transparent. 

AoAs  should  not  focus  on  one  preferred  alternative.  The  preferred  alternative  may  not  be 
attainable  or  viable,  and  a  program  based  on  this  alternative  may  be  canceled  in  later  years. 

•  AoA  should  not  be  focused  on  one  alternative  for  the  solution.  This  one  solution  may 
not  be  attainable  or  viable,  and  a  program  based  on  this  solution  may  be  canceled  in 
later  years. 

•  The  Services  should  not  prematurely  constrain  trade  space.  Locking  down  too  early 
will  restrict  flexibility. 

The  following  items  should  be  included  in  AoAs: 

•  Cost-effectiveness  analyses 

•  Real  world  test  results 

•  Real  world  foreign  alternatives 

•  Early  identification  of  cost  and  perfonnance  drivers 

•  Operational  effectiveness  to  cost  graphs 

•  Sensitivity  analyses  on  the  assumptions 

•  Consideration  of  the  resources  available,  requirements,  and  acquisition  process 

Cost  and  schedule  should  be  considered  when  developing  requirements. 

•  The  JROC  should  consider  costs  when  developing  requirements. 

•  For  the  requirements,  the  “no  kidding”  minimum  values  for  the  requirements  should 
be  identified.  The  requirements  should  be  realistic  so  the  program  is  not  trying  to 
produce  an  unattainable  solution.  The  cost  should  be  considered  when  developing 
these  requirements.  Also,  scheduling  aspects  should  be  considered. 

AoAs  should  be  streamlined  to  reduce  the  cost  of  conducting  the  AoAs. 


120 


APPENDIX  D 

ADDITIONAL  RECOMMENDATIONS  FROM  INTERVIEWS 

This  appendix  provides  additional  recommendations  to  better  meet  OSD  CAPE 
expectations,  better  meet  other  AoA  customer  expectations  and  improve  the  usefulness  of  AoAs. 
These  recommendations  were  taken  from  the  Army  and  OSD  CAPE  personnel  interviews.  The 
five  key  recommendations  are  included  in  Chapter  5. 

General  Recommendations 

•  The  AoA  guidance  should  only  include  three  points:  (1)  the  purpose  for  the  AoA;  (2) 
when  the  AoA  results  are  needed;  and  (3)  identification  of  the  key  angst/friction/ 
concern  points  or  issues. 

•  Ensure  the  AoA  analysts  have  a  better  understanding  of  their  customers,  stakeholders, 
and  external  operating  environment. 

•  AoA  should  not  address  the  acquisition  strategy,  the  number  of  contractors,  how  to 
build  the  system,  or  the  Technology  Demonstration  feeds  into  engineering  design. 

•  Investigate  the  use  of  a  “concept  vehicle”  approach  to  developing  systems  and  a 
“concept  vehicle  show”  for  industry  to  demonstrate  its  technologies. 

Alternative  Related  Recommendations 

•  Perfonn  a  detailed  screening  analysis  to  perfonn  a  board  search  for  alternatives.  The 
screening  analyses  should  start  with  a  broad  look  for  possible  alternatives  and  reduce 
these  alternatives  based  on  criteria. 

•  For  the  MS  A  AoA,  obtain  input  from  industry  as  to  what  technology  or  systems  are 
available. 

•  Limit  the  number  of  alternatives  that  go  through  a  full  suite  of  analyses  to  six  or 
fewer.  Out  of  these  six  full  alternatives,  have  at  most  two  foreign  alternatives. 
Tradeoff,  sensitivity,  and  parametric  analyses  can  be  conducted  on  these  six 
alternatives  to  further  explore  the  trade  space. 

•  The  alternatives  should  be  a  set  of  attributes  and  should  not  be  related  to  a  certain 
system  or  platform. 

•  Identify  an  organization  outside  the  Anny  AoA  community  to  obtain  data  on  foreign 
systems. 


121 


•  Assign  an  Army  PM  to  develop  a  notional  schedule  for  the  foreign  alternative  as 
input  into  the  tradeoff  and  risk  analyses. 

•  The  AoA  should  identify  all  modification  that  would  be  required  for  a  foreign  system 
to  operate  in  the  U.S.  Anny  :  Address  U.S.  standards,  U.S.  GFE  (e.g.,  radios),  or  U.S. 
manufacturing  sites.  Modifications  also  may  be  required  to  meet  desired  capability. 
Include  the  cost,  schedule,  perfonnance,  and  risk  implications  for  these  modifications 
in  the  AoA. 

•  The  AoA  team  should  recommend  to  the  SAG  that  alternatives  be  eliminated  if  they 
are  no  longer  feasible  or  add  nothing  to  the  decision  choices. 

Analyses  Recommendations 

•  For  AoAs  that  have  OSD  CAPE  oversight,  the  Anny  should  ensure  that  all  analyses 
is  quantitative-based  instead  of  qualitative  judgment-based.  OSD  CAPE  is  not  in 
favor  of  SME-based  value  assessments. 

•  The  AoAs  should  focus  on  other  analyses  besides  effectiveness,  including 
sustainment,  force  structure,  transportability,  and  DOTMLPF. 

•  While  conducting  an  AoA,  the  analyses  that  do  not  add  infonnation  relevant  to  the 
decision  should  be  stopped. 

•  The  analyses  should  identify  “tipping  points”  for  the  decision  choices.  If  there  are  no 
tipping  points,  or  points  of  interest,  the  AoA  should  “stop  drilling”  when  the  result 
does  not  make  a  distinction  between  alternatives. 

•  For  the  concept  design,  the  Army  needs  first-order  estimates  that  are  rapidly 
developed  without  the  detailed  design.  Performance  and  cost  estimates  will  have  to  be 
made  for  the  evolving  concept  designs,  which  are  due  to  trades. 

•  AoAs  should  include  affordability-portfolio  assessments. 

Tradeoff  and  Risk  Analyses  Recommendations 

•  Develop  a  coherent,  robust  risk  and  trade  methodology.  The  methodology  must 
address  the  relationship  between  cost,  schedule,  and  perfonnance.  The  interaction  of 
the  tradeoff  and  risk  assessments  also  must  be  developed.  AMSAA  currently  is 
working  on  this  approach,  and  DASA-CE  has  a  tool  to  support  trades. 

•  The  boundaries  for  the  trade  space  should  be  expanded  to  ensure  a  robust  decision 
space. 
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The  trades  should  be  explored  throughout  the  AoA. 

•  Tradeoff  analyses  should  be  conducted  across  and  within  alternatives  and  at  the 
system  and  subsystem  levels. 

•  Frame  the  tradeoff  analyses  for  cost,  schedule,  and  technology.  Technology  is  the 
physical  aspect  of  the  perfonnance  trade. 

•  Risk  assessments  should  be  framed  in  the  context  of  cost,  schedule,  and  technical 
risks. 

•  Risk  methodology  should  have  different  approaches  to  consider  past  systems/ 
components  and  new  systems/components. 

•  Risk  assessment  should  be  performed  on  tradeoffs  between  cost,  schedule,  and 
perfonnance. 

•  Include  operational  effectiveness  to  cost  graphs  and  performance  vs.  cost  graphs  in 
the  tradeoff  and  cost-effectiveness  analyses  results. 

•  The  AoA  should  include  parametric  or  sensitivity  analyses  on  “lesser”  alternatives  to 
detennine  if  they  could  be  made  more  competitive. 

Cost-Related  Recommendations 

•  Use  O&S  and  LCC  in  AoAs,  specifically  for  the  tradeoff  analyses. 

•  Refine  cost  methodology  to  develop  more  estimating  relationships  that  link 
perfonnance  to  cost,  better  incorporate  risk,  and  consider  affordability. 

•  The  Capability  Knowledge  Base  should  be  populated  with  ACAT  II  and  III  data. 

•  The  CAIG  should  be  included  in  the  development  of  cost  estimates. 

Resource-Related  Recommendations 

•  Establish  a  funding  site  for  the  organizations  that  conduct  AoAs  (e.g.,  TRAC, 
AMSAA,  DASA-CE,  and  ARL  SLAD). 

•  Train  additional  analysts  for  technical  risk  assessment,  cost  analyses,  and 
affordability  assessments. 

•  AoAs  should  be  streamlined  to  reduce  the  cost  of  conducting  the  AoAs. 
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